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THE IMPLEMENTATION AND EFFECTS OF THE MATHEMATICS DESIGN 
COLLABORATIVE (MDC): EARLY FINDINGS FROM KENTUCKY NINTH-GRADE 

ALGEBRA 1 COURSES 

Joan L. Herman, Deborah La Torre Matrundola, Scott Epstein, Seth Leon, 

Yunyun Dai, Sarah Reber, and Kilchan Choi 
CRESST/University of California, Los Angeles 

Abstract 

With support from the Bill and Melinda Gates Foundation, researchers and experts in 
mathematics education developed the Mathematics Design Collaborative (MDC) as a 
strategy to support the transition to Common Core State Standards in math. MDC provides 
short formative assessment lessons known as Classroom Challenges for use in middle and 
high school math classrooms. UCLA CRESST’s study of ninth-grade Algebra 1 classrooms 
in Kentucky implementing MDC showed strong support from teachers for the intervention 
and a statistically significant positive impact on student scores on the PLAN Algebra 
assessment, as compared to similar students statewide in Kentucky. 

Chapter 1: 

Introduction 

Kentucky was the first state to adopt new, more rigorous college and career ready 
standards — the Kentucky Core Academic Standards (KCAS) — and commit to statewide 
implementation to ensure that students acquire the knowledge and skills they need for future 
success. There is no doubt that the KCAS substantially raises the bar for expected student 
learning. In mathematics, the KCAS demands that students not only understand mathematics 
concepts and procedures, but also are able to apply and use their knowledge to model and solve 
real-world problems. The result is a set of intellectually demanding standards that call on the 
integration of mathematical content — the basic concepts and procedures of mathematics — with 
mathematical practices — the habits of mind, thinking, reasoning, and modeling which 
characterize mathematics competency. 

KCAS implementation brings with it important challenges for teachers as well as students. 
To prepare students for these new demands, teachers must not only adjust and deepen the content 
of their curriculum and teaching, but also make substantial changes in their pedagogical practice. 
Building students’ capacity to think critically, analyze, communicate, and model and solve novel 
problems in mathematics requires that classroom teachers engage and support students in such 
practices, which is at odds with the teacher-directed instruction and basic mathematical 
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knowledge and procedures foci which have typically dominated mathematics classrooms 
(National Mathematics Advisory Panel, 2008; National Research Council, 2011). 

The Bill and Melinda Gates Foundation invested in the Mathematics Design Collaborative 
(MDC) as one strategy to support teachers’ and students’ transition to more rigorous 
mathematics standards. Although MDC is at a relatively early stage of implementation, the 
Foundation was interested in obtaining early evidence on program effectiveness and contracted 
with the National Center for Research on Evaluation, Standards, and Student Testing (CRESST) 
to conduct a quasi-experimental study of its implementation and learning impact. The study 
examined MDC as it was implemented in ninth-grade Algebra 1 classes from selected Kentucky 
districts during the 2012-2013 school year and is the subject of the current report. 

In the remainder of this chapter, we present background on the study, including a brief 
description of the MDC intervention and the evaluation questions that guided the ninth-grade 
study. In the following chapters, we summarize the study methodology, present the 
implementation and outcome results, and examine the implications of our findings. 

Mathematics Design Collaborative Overview 

MDC supports the transition to the Common Core State Standards (CCSS) by providing 
Classroom Challenges that can be used in conjunction with ongoing curriculum and instruction. 
Each Challenge is designed as a two- to three-day formative assessment lesson (FAL), anchored 
in the Common Core State Standards for Mathematics (CCSS-M), to help secondary 
mathematics teachers monitor and assess their students’ development of key mathematical skills 
and understandings. The Challenges are also intended to model and help teachers incorporate 
deeper mathematical reasoning and thinking into their practice. Towards this end, there are two 
primary types of Challenges, one focusing on conceptual understanding and the other on problem 
solving. At the time of the study, participating teachers could choose from among 40 Challenges 
at the high school level, as well as 6 1 Challenges geared towards middle school mathematics. 

Developed by the Shell Center at the University of Nottingham in collaboration with the 
University of California, Berkeley, each Challenge follows the same general structure: 
(a) Students engage in a pre-assessment, including challenging problems or questions involving 
previously learned concepts or principles, (b) Teachers review student responses to assess 
student approaches, solution strategies, and understandings and misconceptions, (c) Students 
then engage in whole-class and small-group collaborative activities to discuss alternative 
approaches, bring misconceptions to the surface, deepen their understanding, and connect and 
apply their knowledge and skills in new contexts, (d) At the end of the Challenge, students return 
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to the initial problems or questions, revise their initial responses, and reflect on what 
understandings have been gained. 

The Challenges incorporate key changes in pedagogy and classroom culture to support 
CCSS-M learning goals. According to the developers (Math Assessment Project, 2013, p. 2), 
these include the following: 

• Students take more responsibility for their own work. 

• Students engage in “productive struggle” with rich challenging tasks. Resolution comes 
only gradually through interactions and discussion in the lesson as students gain new 
facets of connected understanding. 

• Students study fewer tasks, but in greater depth. They are asked to draft solutions, 
compare their approaches to others, and redraft their ideas as a result of their 
discussions. 

• The teachers’ role is to prompt students to reflect and reason through their ideas. 
Teacher questioning is central to support students’ thinking and depth of knowledge, 
and student growth. The teacher’s role is not to provide answers and solutions. 

Evaluation Questions 

At the time of the study, teachers had had only one or two years of prior experience in 
implementing MDC. These teachers were part of the initial trials of MDC with early district 
implementers. At this early phase of MDC development, the study addressed a comprehensive 
set of evaluation questions: 

1 . How do teachers implement MDC? 

2. What is the impact of MDC on student learning? 

3. What conditions and contexts, including implementation quality, influence MDC 
effectiveness? 

In addressing these questions, the study used a quasi-experimental design (QED) and 
developed and validated new measures of implementation and learning impact, as described in 
the next chapter. 
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Chapter 2: 

Study Methodology 

The study focused on ninth-grade Algebra 1 teachers and their students to examine study 
program effects over the 2012-2013 school year. Study methodology featured a strong quasi- 
experimental design to examine MDC’s effects on students’ state assessment performance, and 
also used implementation and student outcome measures that were specially developed to align 
with MDC goals. The implementation measures included logs, teacher surveys, and analysis of 
student work. Next, we provide details about the study methodology. 

Implementation Measures 

Our implementation measures draw on research on instruction and instructional change, 
given that the ultimate goal of the MDC intervention was to support teachers’ and students’ 
transition to new College and Career Ready Standards (CCRS) that emphasize mathematical 
reasoning and the solving of authentic problems. However, while classroom practice is 
notoriously impervious to refonn (Cuban, 1984; Lortie, 1975), an emerging body of research has 
documented the relationship between student achievement and specific instructional practices 
that create opportunities to learn (see Bryk, Sebring, Allenworth, Luppescu, & Easton, 2010; 
Rowan & Correnti, 2009; Winters & Hennan, 2011). Our implementation measures thus focus 
on classroom instruction, even as we recognize the many influences on the implementation of 
new practices. Study implementation measures included web-based teacher logs, the collection 
and analysis of MDC artifacts, and teacher surveys. When teachers had more than one Algebra 1 
class, they were asked to focus on one class randomly selected by CRESST. 

Web-based teacher log. Study teachers were asked to submit four to six logs over the 
course of the school year. Each log was supposed to be completed during the days directly 
following administration of a Classroom Challenge. The logs focused on (a) the degree to which 
instruction generally aligned with the structure of the MDC intervention and (b) the quality and 
extent of formative assessment practices incorporated into the instruction. The logs were 
structured with conditional branching so that teachers were first given a general set of questions 
about the components of the Challenge they implemented and then were directed to additional 
sets of questions about the pre-assessment, small-group collaborative work, whole-class plenary, 
and post-assessment depending upon their initial responses. Log administration was conducted 
from September 2012 through May 2013. (See Exhibit A1 in Appendix A for a copy of the 
teacher log.) 

During analysis, log data were aggregated by teacher and then were summarized across 
teachers in the sample. That is, item-by-item mean scores were computed for each teacher across 
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all logs submitted. The computations did not include any sections that were skipped because of 
the conditional branching. 

Analysis of MDC artifacts. As part of the log process, teachers were asked to submit 
electronic or hard copies of students’ pre- and post- assessment responses for each Challenge 
completed. Students’ pre- to post-assessment improvement was treated as a measure of teachers’ 
quality of MDC implementation. Our assumption was that teachers who implemented the 
Classroom Challenges well would produce greater student learning improvement than teachers 
whose implementation was less expert. 

Toward this end, three rubrics were developed to examine how students’ mathematical 
learning progressed from pre- to post-assessment for each Challenge. These included rubrics to 
address two positive dimensions of learning, content accuracy and quality of mathematical 
explanations, while a third rubric examined evidence of misconceptions. Accuracy items and 
misconceptions were generally scored on a scale of 0-1. In contrast, explanations were scored on 
a scale of 0-3, with 0 meaning no response, 1 meaning no conceptual understanding, 2 meaning 
partial understanding, and 3 meaning there was evidence of full understanding. (See Exhibit A2 
in Appendix A which includes exemplars and the rubrics used to evaluate student learning on 
each assessment.) 

A subset of Challenges that were commonly used and well aligned to the special CRESST 
mathematics measure were selected for scoring, and individually li nk ed pre- and post-assessment 
scores were used to create a measure of implementation quality. A minimum of 10% of the 
responses for each assessment item were double-coded by specially trained members of our 
research staff, reaching reliability of > .80 per item and > .90 per Challenge. Our assumption was 
that greater pre-to-post learning progress could serve as an indicator of higher quality MDC 
implementation. 

Teacher survey. CRESST collaborated with Research for Action (RFA) on the design of 
the spring 2013 implementation and scale-up survey for MDC teachers. The survey included a 
section on module implementation with items designed to mirror the intent of the log items, and 
to augment information on factors likely to influence MDC implementation. As such, the survey 
emphasized implementation of the Challenges during the 2012-2013 school year; use of the 
Challenges during previous school years; support received for implementation (e.g., professional 
development, collaboration, etc.); and teacher perceptions about the Challenges, their impact on 
students, and mathematics skills in general. We drew on RFA survey variables as indicators of 
implementation quality as well as context and possible moderators of MDC implementation and 
impact (e.g., experience using MDC, attitudes regarding mathematics instruction, extent of 
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professional development, leadership support, and collaboration). Descriptive statistics were 
computed at the teacher level. (See Exhibit A3 in Appendix A for a copy of the teacher survey.) 

Post-intervention opportunity-to-learn survey. Participating teachers were asked to 
complete a short one-page survey at the time that they administered the CRESST-designed 
mathematics assessment. This survey included questions on the administration of the 
assessments (i.e., the administration date and amount of time students spent completing them) as 
well as several questions about the Challenges implemented during the 2012-2013 school year 
(i.e., number taught and topics covered). The survey also asked teachers to report on the degree 
of emphasis they placed in their 2012-2013 instruction on the content areas covered by the 
CRESST mathematics assessment. The answers to these questions were intended to provide 
valuable data on students’ opportunity to learn (OTL) the content tested and to help tease out the 
effects of OTL and MDC instruction. Although results were summarized at the teacher level, 
because of low return rates the data were of limited utility. (See Exhibit A4 in Appendix A for a 
copy of the OTL survey.) 

Student Outcome Measures 

Study outcome data included individual student-level achievement data from Kentucky’s 
state assessment program and the special CRESST measure. The following describes each of 
these in turn. 

Longitudinal state assessment data. All Kentucky students take ACT’s PLAN test in the 
fall of 10th grade. The mathematics component of ACT PLAN is a 40-minute multiple choice 
exam, including 22 items addressing algebra and pre-algebra content and 18 items addressing 
geometry. The exam yields subscores for each of these areas. Score reliabilities for the subscores 
and total scores ranged from .65 to .86, with a reliability of above .80 for algebra, the primary 
area of interest in the current study (ACT, 2013). 

Tenth-grade ACT PLAN scores from the fall of 2013 for students participating in 
Algebra 1 during the 2012-2013 academic year served as the outcome measure for the study’s 
QED analyses. In addition, the study gathered prior years of Kentucky state assessment scores 
going back to 2008-2009 for study students, teachers, and schools. These data were used in 
Coarsened Exact Matching (CEM) to create a comparison group that was carefully matched to 
the MDC group in student and school characteristics, prior achievement, and prior teacher and 
school effectiveness. Further details about the matching are presented in Chapter 4. 

CRESST mathematics assessment. CRESST developed and administered a mathematics 
learning measure to students in participating MDC teachers’ classrooms (see Exhibit A5 in 
Appendix A). This measure was aligned with the conceptual content of the five Challenges most 
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commonly implemented by study participants during the 2012-2013 school year. The measure is 
composed of a variety of item types that, as described in Table 1, address different kinds of 
knowledge. 


Table 1 


Item Types and Knowledge Targets for CRESST Mathematics Assessment 


Item type 

Description of knowledge targets 

Basic computation 

Respond to a simple, well-defined problem, e.g., a computation problem that requires 

(BC) 

finding an answer or missing variable. 

Knowledge required is recall of concepts or procedural rules or principles. 

Problem represents an application of the relevant big idea or key concept. 

Word problem (WP) 

Generate a written solution or product in response to a problem-solving prompt. 

Problem solution requires application of one or more big ideas and integration of 
declarative (factual), conceptual, and procedural knowledge. 

Problem situation is described in text or represented graphically, followed by a short (1-2 
sentence) question or prompt. 

It should be possible to read the directions and complete the task in 5 minutes. 

Word problem with 

Generate a written solution or product in response to a problem-solving prompt, and 

justification (WJ) 

explain the what and why of the solution. 

Problem solution requires application of one or more big ideas and integration of 
declarative (factual), conceptual, and procedural knowledge. 

Problem situation is described in text or represented graphically, followed by a short (1-2 
sentence) question or prompt. 

It should be possible to read the directions and complete the task in 5-7 minutes. 

Explanation (EX) 

Generate a clear, coherent explanation of a big idea and indicate how it can be used to 
solve problems. 

Students must support their explanations with examples and illustrations. 

Directions may provide an authentic context to frame the writing, consisting of the 
intended audience and a situation or scenario to which the student should respond. 


The assessment included five sections, or mini-tests, aligned to the content of the following 
Challenges most commonly administered by participating ninth-grade Algebra 1 teachers: 
(FAL03) Solving Linear Equations in Two Variables, (FAL13) Sorting Equations and Identities, 
(FAL16) Interpreting Algebraic Expressions, (FAL20) Forming Quadratics, and (FAL22) 
Finding Equations of Parallel and Perpendicular Lines. Each mini-test was composed of five or 
six items. 



Rubrics and scoring keys were developed by CRESST in order to score each assessment 
item. All multiple choice and short answer (fill in the blank) items, including basic computation 
and word problems, were scored using a 0-1 scale (0 = incorrect, 1 = correct). In contrast, a 
scale of 0-2 was used to score extended response items involving justification and/or 
explanations (0 = incorrect, 1= partially correct, 2 = correct). Exhibit A6 in Appendix A 
contains the rubrics used to score the explanation items on the CRESST math assessment. 

Data on score reliability are shown in Table 2. As would be expected, reliability was 
highest for the overall assessment (a = .82). In contrast, reliability of the mini-tests addressing 
each commonly used Challenge ranged from .35 to .64, with all but one above .52. While these 
results are not surprising given the small number of items constituting each mini-test, they do 
argue against the use of mini-test scores in subsequent analyses. Interestingly, the reliability of 
the extended response items (a = .67) was also somewhat lower than the reliability for the 
multiple choice and short answer items when collapsed together (a = .73). Because of the 
reliability results for the mini-tests, descriptive analysis was limited to overall total scores and 
scores by item type. 


Table 2 


Reliability Statistics for CRESST Mathematics Assessment 


Section 

Number of items 

Cronbach’s alpha 
(reliability) 

Results by mini-test 

Solving Linear Equations in Two Variables (FAL03) 

5 

.59 

Sorting Equations and Identities (FAL13) 

6 

.55 

Interpreting Algebraic Expressions (FAL16) 

5 

.35 

Forming Quadratics (FAL20) 

6 

.52 

Finding Equations of Parallel and Perpendicular Lines (FAL22) 

6 

.64 

Results by item type 

Extended response items 

7 

.67 

Multiple choice and short answer items together 

21 

.73 

Overall 

28 

.82 


Study Sample 

Population. The population of interest was all Algebra 1 teachers (and their students) in 
the six districts and 17 schools across the state of Kentucky that were the earliest adopters of 
MDC. This included the teachers who participated in the Phase 1 MDC implementation in 2010- 


9 



2011 or its Phase 2 expansion in 2011-2012, and who continued to teach Algebra 1 in the 2012- 
2013 school year. We focused on ninth-grade Algebra 1 teachers because of (a) their relative 
concentration in the larger population of teachers participating in MDC, (b) the availability of a 
statewide Algebra 1 measure that could serve as an outcome measure for the study, and (c) the 
importance of Algebra 1 performance in students’ subsequent academic success. 

All MDC teachers were required by their districts to implement at least four to six 
Challenges over the course of the study year (2012-2013), and in some cases implemented many 
more. While some districts integrated specified Challenges within their pacing plans and teachers 
were required to follow these plans, in other districts the choice and timing were at the discretion 
of the individual teachers. 

Study completion rates for MDC-only measures. Schools with teachers who were 
implementing MDC in ninth-grade Algebra 1 were invited to participate in the study. Once 
district and principal pennissions were obtained, district leadership and project coordinators 
provided CRESST with support in determining eligibility, and provided contact information for 
those considered eligible. All participation was voluntary and varied across districts and schools. 
Based on information received from the districts, and subsequent communication with teachers, 
we estimated that there were 46 teachers eligible for the study. As shown in Table 3, nearly two 
thirds of all eligible MDC teachers consented to participate in the CRESST implementation 
study, and over three quarters completed the RFA survey. Almost all of the teachers who 
consented for the study also turned in artifacts, completed logs, and administered the CRESST 
mathematics measure to their students. In contrast, less than half of consented teachers 
completed the OTL survey. 
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Table 3 


MDC Study Completion Rates 


MDC teachers (ninth grade, Algebra 1) 

n 

% relative to 
all eligible 
MDC teachers 

% relative to 
consented 
teachers 

Eligible teachers 

46 

-- 

-- 

Teachers consenting to CRESST implementation study 

30 

65 

- 

Logs 

27 

59 

90 

Teacher artifacts 

28 

61 

93 

CRESST mathematics measure 

26 

56 

87 

OTL survey 

12 

26 

40 

Teachers completing RFA survey 

36 

78 

- 


Sample for quasi-experimental design. All MDC teachers in the study population and 
their students during the 2012-2013 school year were included in the QED study of MDC effects 
on student learning. Collection of linked teacher and student data from school district points of 
contact yielded a larger group of eligible teachers than originally identified for participation in 
the MDC-only implementation measures: 97 teachers in total. However, after removing teachers 
who did not teach Algebra 1 classes according to received state databases, only 59 teachers and 
their students remained for the analysis. Given the importance of sample size to our analyses, we 
erred on the side of inclusiveness, and relied on this larger sample for the QED study rather than 
the smaller sample from the implementation measures. Using longitudinal student and teacher 
data from Kentucky’s state database, and drawing on ninth- grade Algebra 1 students in similar 
courses across the state, we used CEM to create a comparison sample of students who were 
nearly identical to the MDC group in demographics, prior academic performance, and selected 
school and district characteristics. Comparison students needed to have an ACT PLAN outcome 
score to be eligible for matching. Because treatment students and teachers were not randomly 
selected to participate in the MDC initiative, it was necessary to use a matching technique, such 
as CEM, to control for the effects of student, teacher, and school characteristics that might 
influence student outcomes. The CEM process led to the loss of 374 students and three of the 59 
teachers. 

Table 4 shows the demographic and prior achievement characteristics of the MDC and 
comparison student samples for the QED design, based on available state data. These data 
indicate that the study’s student population was predominantly White, with a nearly equal 
representation of males and females. Nearly half of the students qualified for free or reduced 
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price lunch. Although just over 10% of the students were identified as students with disabilities, 
very few were classified as limited English proficient (LEP). 

Table 4 


Demographic Characteristics of Kentucky Students in Eligible and Matched Samples for QED Study 


Student characteristics 

Eligible sample 

Matched sample 

MDC 
(n = 2,690) 

Comparison 
(n = 28,892) 

MDC 
(n = 2,316) 

Comparison 
(« = 15,325) 

Prior achievement 

0.056 

-0.005 

0.090 

0.065 

Female (%) 

49.6 

50.2 

48.5 

48.6 

Ethnicity 





White (%) 

86.1 

82.4 

92.4 

90.3 

Hispanic (%) 

4.0 

3.2 

1.7 

2.1 

Black (%) 

6.1 

11.7 

3.9 

5.0 

Asian (%) 

1.0 

0.6 

0.4 

0.7 

LEP (%) 

2.3 

1.4 

0.5 

0.8 

Special education (%) 

11.8 

11.6 

13.6 

12.0 

Free or reduced price lunch eligible (%) 

48.5 

61.7 

56.3 

53.4 


Note. Prior achievement scores are standardized. 


As can be seen in Table 4, matched MDC and comparison student samples were very 
similar in demographic makeup and mean prior achievement. 
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Chapter 3: 

MDC Implementation 

In this chapter, we present descriptive findings from our implementation measures, 
including teacher logs, surveys, and artifacts from the Classroom Challenges. In reviewing these 
findings, it is important to keep in mind the small sample sizes on which results are based, and 
that while logs, surveys, and artifacts overlap, they are not fully the same. For example, some 
teachers completed the survey but did not complete the log and vice versa (see Table 3 in 
Chapter 2). Tabulated results for the teacher log and teacher survey can be found in Appendices 
B and C respectively. 

Teacher Background 

Teachers’ background, prior experience, and attitudes about mathematics instruction, 
gleaned from teacher survey responses, provide important context for the implementation 
findings. Survey responses indicate a wide range of experience among participating Algebra 1 
teachers (see Table C2 in Appendix C). On average, teachers reported having taught for nearly 9 
years, with some having only 2 years of experience and others up to 21 years. A majority of the 
average teacher’s experience was within her current district (M = 6.96 years) and within her 
current school (M = 5.86 years). At the time of the survey, all but one of the participants was 
currently teaching Algebra 1, with many also teaching other mathematics subjects, for example, 
geometry or Algebra 2 (Table Cl). Almost all teachers reported that they had at least some 
students who were classified as special education or who were struggling in mathematics (Table 
C3). About two thirds of the teachers also indicated that they had students who were English 
language learners (ELL) and/or in advanced mathematics. 

Teachers also varied in whether or not their participation in the initiative was mandatory, 
with about two thirds stating that they volunteered to participate (Table C4). On average, the 
teachers who completed the survey reported a small increase in the number of Challenges they 
implemented from approximately five during the 2011-2012 school year to six in the 2012-2013 
school year (Table C5). This increase is not surprising in that most teachers agreed at least 
somewhat that the Challenges were aligned with their school curriculum and the Common Core, 
and prepared their students for the state assessments. These teachers also tended to report that 
they saw the value of MDC for addressing the CCSS (Table C6). 

Teacher Logs 

As previously indicated, participating teachers were asked to complete logs after four to six 
Challenges. Of the teachers who completed logs, two thirds (n = 18) completed at least four and 
approximately one fifth (n = 6) submitted six or more. Among the teachers who did not meet 
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these thresholds, about half ( n = 4) consented for the study approximately one third of the way 
into data collection and thus likely completed Challenges prior to their study participation. 

Teachers tended to follow guidelines concerning the length of time allocated for each 
Challenge, spending an average of approximately two class periods or 100 minutes on each. 
Furthermore, teachers reported spending about a week of classroom time on the mathematics 
concept(s) central to each challenge prior to their implementation (see Table B1 in Appendix B). 
There was, however, wide variability in the amount of prior instruction, ranging from one half to 
15 class sessions on Challenge concepts (M = 4.64, SD = 4.01). Furthermore, teachers reported 
that they mostly followed the detailed lesson plans for the Challenges they implemented (Table 
B2). 

Considering the variability in time allotted to different major and minor components in the 
teacher guides, it was not surprising to find variability in the times teachers reported in their logs 
(Table B3). In general, teachers spent an average of about one third of their Challenge time 
engaging students in small-group collaborative work and about one fifth with whole-class 
discussion. Pre- and post-assessments as well as teacher-led review of content and concepts each 
averaged about 10% of the time per Challenge. Consistent with guidelines, little time was spent 
on individual student work or student presentations. 

Turning to the pre-assessment component, teachers generally asked students to complete 
the assessment prior to the first day of implementing the Challenge (Table B4). As with other 
implementation questions, teachers showed wide variation in the time they spent reviewing 
students’ responses to the pre-assessment (Table B5). Over half (57%) spent 16 or more minutes 
on their review, with the modal response of 16-30 minutes spent in review (34%). Teachers most 
typically reported reviewing their students’ pre-assessment responses alone (49%) or with a 
colleague (38%). Very few teachers reviewed the answers with their class (Table B6). 
Whichever the context, teachers reported reasonable confidence in their understanding of their 
students’ misconceptions based on their review of the pre-assessment responses (Table B7). 

In contrast to other log responses that showed close fidelity of implementation, teachers’ 
responses showed some deviation relative to the feedback they provided students about their pre- 
assessment responses. Over half of the teachers (58%) reported reviewing students’ common 
errors at the start of class, with the next most popular form of feedback being commenting on 
papers (14%). At the same time, many teachers (46%) reported providing no feedback since they 
felt the Challenge lessons addressed the misconceptions, and very few teachers wrote questions 
on individual students’ papers (Table B8). These findings stand somewhat in contrast to 
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Challenge guidelines, which suggested following up on misconceptions via whole-class probing 
or writing questions on individual students’ pre-assessments. 

Teacher reports showed more fidelity to Challenge guidelines in providing feedback during 
small-group collaborative work (Table B9). When teachers observed students struggling in their 
small groups, most reported that they raised questions (93%) or asked another student to explain 
a concept (78%). Although there was a great deal of variation among teachers, on average 45% 
noted that they would offer hints and suggestions, 41% stated that they would listen but not 
intervene, and 30% noted that they would stop the group work to review a concept with the 
whole class. In addition, almost no teachers reported providing struggling students with the 
answer. 

Similarly, teachers’ log responses showed substantial compliance with Challenge 
guidelines in implementing questioning during whole-class plenary sessions (Table BIO). 
Teachers uniformly reported asking students to self-reflect (e.g., Why do you think that?) or to 
reflect on the work of their other classmates (Do you agree?). Furthermore, over three fourths of 
the teachers reported asking students about their mathematical reasoning (84%) and/or to find 
patterns or make conjectures (77%). At the same time, a majority of teachers (65%) employed 
the Initiate-Respond-Evaluate (IRE) technique. This finding seems inconsistent with Challenge 
pedagogy since in IRE, teachers provide feedback on the correctness of the response, pose yes- 
no questions, or encourage students to recall facts or definitions. Teachers also reported that they 
asked relatively few of their students (5-25%) to present their solutions or to articulate their 
reasoning to the whole class (Table B1 1). 

Finally, most of the teachers (82%) reported having their students complete the post- 
assessment in class, at the end of the Challenges (Table B12). In evaluating student success, 
teachers typically reported that between 25% and 50% of their students struggled with central 
concepts during small-group collaboration and that only about 50% of their students had a strong 
grasp of the target concepts by the end of the Challenge (Tables B13 and B14). These results 
suggest that teachers and students may have needed additional support to increase the 
effectiveness of the Challenges. 

Teacher Survey 

In this section, we present descriptive results from the RFA/CRESST teacher survey, 
concentrating on the subset of items most relevant to the implementation study. This includes 
items developed to parallel questions in the teacher log as well as a series of questions about 
teachers’ sense of efficacy for teaching MDC, school and district support for the initiative, extent 
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of informal teacher collaboration and formal professional development, and teacher perceptions 
of the impact of the intervention on students. 

When asked about their implementation of the MDC lessons, most teachers (94%) reported 
that they mostly or completely adhered to the teacher guides (Table Cl). On average, results 
showed that teachers generally followed or exceeded recommended lengths of time for the 
Challenges, 1 with a majority indicating that they spent three class periods on each (Table C8). 
Even so, while the vast majority of teachers (89%) reported that their students completed the 
post-assessment for their most recent Challenge, two teachers did indicate that they could not 
because they ran out of time (Table CIO). In implementing the Challenges (Table C9), over half 
of the teachers assigned students to homogeneous collaborative groups (57%), while the others 
assigned heterogeneous groupings (20%) or let students group themselves (23%). 

Teachers were also asked to share their perceptions about the effectiveness of various 
pedagogical strategies during Challenges (Table Cll). They agreed most strongly about the 
effectiveness of having students ask guiding questions and about teachers acting more as 
facilitators or coaches. Teachers also tended to agree that peer-to-peer problem solving and 
giving students time to persevere through difficult problems benefited student understanding of 
mathematics. 

Teachers felt that the MDC lessons benefited their teaching in a number of ways (Tables 
C12 and C13). Most prominent among these were teachers’ beliefs that the lessons provided 
effective strategies for teaching mathematics (89%), taught them new formative assessment 
strategies (89%), helped them promote mathematics discourse (86%), and helped them detennine 
their students’ strengths and weaknesses (81%). Although still noted by two thirds of the 
teachers (69%), the least cited benefit involved being able to provide students with more detailed 
feedback about their work. Teachers typically agreed that MDC was a useful resource for 
addressing the CCSS and helped them to use formative assessment to identify student strengths 
and weaknesses and inform their instruction. Teachers also generally felt that their MDC 
participation encouraged them to focus on building conceptual knowledge rather than on process. 

It is also interesting to note that teachers tended to agree somewhat that they had sufficient 
time to prepare for the Challenges and for the pre-assessment specifically, and that they were 
adequately prepared to effectively use the lessons. During the Challenges, teachers also tended to 
feel they were able to interact with all of the collaborative groups in their classes (Table C14). 


'Recommended times for the Classroom Challenges teachers reported implementing ranged from 75 to 120 minutes. 
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When reporting on administrator support and collaboration with colleagues, over two thirds 
noted that other teachers (94%), their department head (77%), and/or their district or network 
MDC lead (71%) visited their classroom during the MDC lessons. In contrast, less than half 
(40%) reported that their principal observed their MDC teaching (Table Cl 5). Similarly, teachers 
tended to disagree that their school administrator provided them with feedback about their MDC 
instruction or attended MDC professional development. Despite this, teachers overall agreed 
somewhat that their school administrator encouraged them to participate in MDC and 
emphasized formative assessment at the school. Interestingly, teachers felt somewhat more 
strongly about district support for MDC (Table Cl 6). 

In addition to having their colleagues visit their classroom, 60% of the teachers noted that 
they had regularly scheduled planning time with other MDC teachers and two thirds reported that 
they met at least once per month (Tables C17 and Cl 8). On average, teachers perceived many 
benefits from peer collaboration (Table Cl 9), indicating most prominently that it helped them 
determine where to use MDC within their existing mathematics units, effectively use the 
formative assessment strategies emphasized in MDC, develop feedback questions, and identify 
common mathematics misconceptions. Collaboration with their colleagues was least helpful, as 
rated by teachers, in determining how to group students for instruction. 

Three fourths of the teachers reported participating in MDC professional development 
during the 2012-2013 school year (Table C20). On average, teachers participated in three 
sessions, with some attending as few as one session and others participating in as many as eight 
sessions (Tables C21 and C22). Teachers most frequently participated in sessions focused on 
developing feedback questions (100%), identifying misconceptions through analysis of the pre- 
assessment results (96%), and facilitating small-group work (93%). The least common topics 
focused on how to implement MDC with special education students (33%) or with ELL students 
(15%). 

Teachers also reported on the format of their professional development and their 
perceptions about the effectiveness of the format (Table C23). Over four fifths of the teachers 
noted that their professional development involved small-group meetings (93%), schoolwide 
meetings (85%), districtwide meetings (85%), and/or coaching (82%). Of these most popular 
formats, teachers found the small-group meetings and coaching sessions most effective (100%). 

Linally, teachers were asked their opinion about the impact and effectiveness of the MDC 
Challenges (see Tables C13 and C24 through C26). Interestingly, while on average, teachers at 
least agreed somewhat that MDC helped them make instruction more engaging, only about one 
third (39%) reported that their students were more engaged during MDC than during other 
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instruction and half (50%) indicated that engagement was the same. Nevertheless, teachers 
tended to agree at least somewhat that MDC improved students’ mathematical reasoning and 
supported students’ college readiness. Finally, most teachers (83%) felt that participation in 
MDC helped most of their students improve their content knowledge and/or conceptual 
knowledge. 

Analysis of MDC Student Work Artifacts: Pre-to-Post Evidence of Learning 

As noted earlier, student work was collected as part of the log process and used to create 
indicators of implementation quality. Individually linked pre- and post-assessments were scored 
using rubrics to examine the quality of responses and evidence of misconceptions. The logic of 
using student responses to judge teacher implementation derived from our assumption that well- 
implemented Classroom Challenges would lead to higher gains in the quality of responses from 
pre- to post-assessment, as well as a reduction in common misconceptions. While teachers 
submitted class sets for 18 of the Challenges, our analyses focused solely on four of the most 
commonly administered by participating ninth-grade Algebra 1 teachers. Furthermore, of the 28 
teachers who submitted any student work for these four, only 29% to 57% submitted matched 
pre- and post-assessment class sets for each of these Challenges (Table 5). Because of this lack 
of representativeness within and across Challenges, results should be interpreted with caution. 

Table 5 


Number of Classroom Sets of Student Work Analyzed by Classroom Challenge 



Pre- and/or post-assessment 

Matched pre-post 

Challenge 

Class sets (n) 

Teachers (%) 

Class sets ( n ) 

Teachers (%) 

Solving Linear Equations in Two Variables (FAL03) 

17 

60.7 

16 

57.1 

Sorting Equations and Identities (FAL13) 

16 

57.1 

16 

57.1 

Interpreting Algebraic Expressions (FAL16) 

13 

46.4 

13 

46.4 

Finding Equations of Parallel and Perpendicular Lines 
(FAL22) 

11 

39.3 

8 

28.6 

Total (one or more common Challenge) 

25 

89.3 

25 

89.3 


Note, n of teachers = 28. 


Table 6 presents descriptive statistics for the pre- and post- assessments that were li nk ed for 
individual students. Although results were generally low, considerable variation was found. On 
the pre-assessments, the average student only earned between 23% and 35% of possible points 
on the explanation items and between 21% and 63% of the possible points on the accuracy items. 
Furthermore, students received an average misconceptions score of between 43% and 81% on 
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their pre-assessments. Note that the misconceptions scores are reverse coded; that is, a higher 
misconceptions score in Table 6 indicates a lower presence of misconceptions. Post-assessment 
performance also showed variation across Challenges. More specifically, average scores on the 
post-assessments ranged from 29% to 43% on the explanation items, from 27% to 72% on the 
accuracy items, and from 38% to 89% on the misconceptions scores. 

Part of the explanation for this variability in performance at both time points may involve 
differences in the focus for each of these Challenges (see Appendix D). For example, FAL16, on 
which the average student tended to score higher, focused on algebraic expressions, a content 
area initially introduced in middle school, while FAL22, with which students tended to most 
struggle, focused on high school standards involving geometric theorems and formal 
representations of functions such as y = mx + b and graphs. In contrast, both FAL03 and FAL13 
focused on linear equations, content that has been traditionally taught in Algebra 1, but with 
which many students still struggle. 

Table 6 


Mean Score Percentages for Individual Classroom Challenges by Rubric Type and Assessment 



n 

Accuracy 
M(SD ) 

Explanation 

M(SD) 

Misconceptions 

M(SD) 

Solving Linear Equations in Two Variables (FAL03) 

Pre-assessment 

307 

.45 (.28) 

.23 (.18) 

.69 (.19) 

Post-assessment 

307 

.53 (.29) 

.29 (.21) 

.72 (.18) 

Sorting Equations and Identities (FAL13) 

Pre-assessment 

96 

.37 (.18) 

.30 (.18) 

.81 (.15) 

Post-assessment 

96 

.58 (.19) 

.43 (.20) 

.89 (.11) 

Interpreting Algebraic Expressions (FAL16) 

Pre-assessment 

96 

.63 (.18) 

.35 (.27) 

.50 (.25) 

Post-assessment 

96 

.72 (.16) 

.41 (.27) 

.61 (.26) 

Finding Equations of Parallel and Perpendicular Lines 
(FAL22) 

Pre-assessment 

56 

.21 (.13) 

.23 (.21) 

.43 (.28) 

Post-assessment 

56 

.27 (.17) 

.33 (.23) 

.38 (.28) 


Note. For this analysis, misconceptions were reverse coded so that higher means indicate a lower presence of 
misconceptions. 


Less variability was found when examining pre- to post-assessment changes in 
performance (Table 7). More specifically, average student accuracy and explanation scores 
increased by only 6-9% for three of these Challenges (i.e., FAL03, FAL16, and FAL22). In 
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contrast, FAL13, which was the only Challenge to include multiple choice items, showed an 
average improvement of 21% on the accuracy items and 14% on the explanation items. At the 
same time, FAL13 showed little improvement (8%) in regards to evidence of common 
misconceptions, which may indicate improvement in computation rather than in conceptual 
understanding of algebra. It also should be noted that FAL22 was the only Challenge to show a 
slight average increase in misconceptions pre- to post-assessment. 

Collectively the findings indicate limitations in the quality of MDC implementation in that 
students did not evidence strong improvements in learning from pre- to post-assessments, nor did 
they demonstrate solid mastery of Challenge goals. 

Table 7 


Mean Percentage Change in Performance for Individual Classroom Challenges by Rubric Type 




Accuracy 

Explanation 

Misconceptions 


n 

M(SD) 

M (SD) 

M (SD) 

Solving Linear Equations in Two Variables (FAL03) 

307 

.08 (.35) 

.06 (.22) 

.02 (.24) 

Sorting Equations and Identities (FAL13) 

96 

.21 (.21) 

.14 (.24) 

.08 (.17) 

Interpreting Algebraic Expressions (FAL16) 

96 

.09 (.22) 

.06 (.31) 

.11 (.34) 

Finding Equations of Parallel and Perpendicular Lines 
(FAL22) 

56 

.06 (.17) 

.09 (.23) 

-.05 (.29) 


Summary of Implementation Data 

Results from teacher logs, surveys, and the analysis of student work from commonly taught 
Challenges provide at least one consistent finding: Across all sources, the data show substantial 
variation across teachers in all aspects of MDC implementation. This included how much prior 
instruction students received on Challenge concepts, how teachers allocated instructional time 
across the various Challenge components, the pedagogical strategies teachers most used, the 
specific strategies they used to formatively assess and provide students with feedback on their 
learning, and their opinions about MDC effectiveness. The analysis of student work during the 
Challenges also showed substantial variation across teachers in the quality of implementation. 

With this variation as a caveat, the findings provide a portrait of how and with what 
support MDC was implemented by the study sample, and participating teachers’ impressions of 
effectiveness. 

Who. Study teachers showed a range of teaching experience, with an average of about nine 
years. Most had spent the bulk of their careers in their current districts and schools. Nearly two 
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thirds indicated that they had volunteered, rather than had been required, to work with the MDC 
tool during the 2012-2013 school year. Most teachers had only one year of experience 
implementing MDC beyond their initial training year. On average, study teachers implemented 
six Challenges during the study year and five during the prior year. 

How. Nearly all teachers reported that they mostly or completely adhered to the MDC 
guidelines in implementing the Challenges, which typically took two to three class sessions for 
them to complete. Teachers, however, varied considerably in the time they spent analyzing 
students’ pre-assessments and in the time and specific strategies they used in implementing each 
component (i.e., the pre-assessment, small-group collaborative activity, whole-class discussion, 
and post-assessment). Although teachers tended to deviate from guidelines in the feedback they 
provided to students about their pre-assessment performance, their reported interactions during 
small-group and whole-class activities were in accord with MDC’s productive struggle 
philosophy. Teachers raised questions, asked students to explain their reasoning and solicit 
feedback from peers, and asked students to self-reflect rather than provided them directly with 
answers. 

Analyses of student performance on the Challenges, as judged by evidence of student 
learning from pre- to post-assessment, however showed a somewhat different picture of 
implementation fidelity. In general, these analyses showed that teachers had difficulty helping 
students to achieve the Challenge goals. Results indicate that improvements in student 
understanding were generally scant and evidence of misconceptions remained. Students appeared 
to particularly struggle with providing explanations of their reasoning. 

With what support. Survey responses indicated that teachers felt strong support for MDC 
from district leadership, but school-level principal support was less consistent across the sample. 
Three fourths reported participating in professional development to prepare for MDC 
implementation and those who did found it beneficial. Although formal time for collaborative 
planning and feedback varied across the sample, the great majority of teachers reported meeting 
informally at least every other week to discuss their MDC work. Teachers also reported that they 
found their peers to be highly collaborative and the collaboration helpful to their MDC 
implementation. 

Attitudes toward MDC. Teachers were very positive about key MDC pedagogical 
strategies such as teacher as facilitator, asking guiding questions, and peer-to-peer problem 
solving, and they found the small-group and plenary approaches helpful to student learning. 
Although they felt that the Challenges benefited students’ conceptual understanding and 
mathematical thinking, they reported that sizable proportions of their students struggled during 
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the Challenges and failed to achieve a firm grasp of the intended target — on average they felt that 
only about half of their students reached this level. As with the MDC findings, these results 
suggest that participating teachers may have needed additional help with the implementation of 
the modules and with their MDC instruction. 
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Chapter 4: 
Student Outcomes 


The study used multiple measures of student learning to examine both the effects of MDC 
and the relationships between the MDC implementation variables and student outcomes. In this 
chapter, we first provide descriptive results for the CRESST mathematics measure and the ACT 
PLAN state assessment for the MDC sample only, followed by the results of the QED analysis of 
MDC effects on learning in Kentucky. 

Descriptive Results 

CRESST mathematics assessment. As noted earlier, CRESST specially developed a 
mathematics assessment to align with the five most commonly implemented Classroom 
Challenges. The assessment included five or six items on each topic, and was composed of 
multiple choice, short answer, and extended response explanation items. Following preliminary 
analyses, results for the multiple choice and short answer items were collapsed together (Table 
8). 

The mathematics assessment was administered in 27 study classrooms, with 47 1 students 
completing the instrument. Although results revealed considerable variation in perfonnance 
across students, the generally low level of perfonnance was clear. On average, students earned 
26% of the possible points on the extended response questions, and 30% of the possible points 
on the multiple choice and short answer questions combined. Overall, students generally earned 
only 29% of the total possible points on the exam. 

Table 8 


Descriptive Statistics for CRESST Mathematics Assessment 


Type 

n 

Total possible score 

Mean 

SD 

Minimum 

Maximum 

Extended response 

471 

11 

2.83 

2.26 

0 

10 

Multiple choice and short answer 

471 

22 

6.71 

3.62 

0 

18 

Overall 

471 

33 

9.55 

5.39 

0 

28 


Part of the explanation for this low performance may reside in students’ opportunity to 
engage in the Challenge topics addressed by the measure. As was previously mentioned, since 
the specific Challenges that were implemented as part of MDC were at the discretion of 
individual districts, schools, and/or teachers, there was no common set in which all students were 
engaged. CRESST used log responses to ascertain the five most commonly used Challenges, 
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which were then used to ground the tested constructs. It should also be noted that while we 
sought to confirm student exposure to these Challenges through the post-intervention OTL 
survey, only 12 of the 26 teachers completed the instrument. Because of this, the results should 
be interpreted with caution. 

Of the teachers who completed the survey, at least 50% reported that their students had 
been engaged with three of these Challenges. In contrast, the remaining two Challenges were 
addressed in less than one third of these study classes (Table 9). 

Table 9 


Teacher-Reported Coverage of Classroom Challenges Used to Ground CRESST Mathematics 
Assessment 


Classroom Challenges 

n 

MDC teachers % 

Finding Equations of Parallel and Perpendicular Lines (FAL22) 

4 

33.3 

Forming Quadratics (FAL20) 

3 

25.0 

Interpreting Algebraic Expressions (FAL16) 

6 

50.0 

Solving Linear Equations in Two Variables (FAL03) 

10 

83.3 

Sorting Equations and Identities (FAL13) 

8 

66.7 


Note, n = 12. 


In addition, teachers who completed the survey were asked to report on the level of 
emphasis they placed on the math standards addressed by the assessment. As seen in Table 10, 
the great majority of the teachers who responded to the survey reported giving each assessed 
standard at least moderate attention during the school year, except for “use coordinates to prove 
simple geometric theorems algebraically.” Based on these results, it seems likely that most 
students in the study had at least moderate engagement with most of the test content. 
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Table 10 


Teacher-Reported Degree of Emphasis Placed on Standards During 2012-2013 School Year 


Standards 

n 

No emphasis 

Slight 

emphasis 

Moderate 

emphasis 

Sustained 

emphasis 

a. Interpret the structure of expressions 

ii 

0.0(0) 

0.0(0) 

72.7(8) 

27.3 (3) 

b. Rewrite rational expressions 

ii 

9.1(1) 

18.2 (2) 

72.7 (8) 

0.0 (0) 

c. Write expressions in equivalent forms 
to solve problems 

12 

0.0 (0) 

8.3(1) 

50.0 (6) 

41.7 (5) 

d. Create equations that describe 
numbers or relationships 

12 

0.0 (0) 

0.0(0) 

33.3 (4) 

66.7 (8) 

e. Solve systems of equations 

12 

0.0(0) 

0.0(0) 

25.0(3) 

75.0(9) 

f. Use coordinates to prove simple 
geometric theorems algebraically 

11 

27.3 (3) 

63.6(7) 

9.1(1) 

0.0 (0) 

g. Analyze functions using different 
representations 

11 

0.0 (0) 

27.3 (3) 

54.5 (6) 

18.2 (2) 


ACT PLAN results. As described earlier, the study drew on students’ performance on the 
ACT PLAN assessment administered to study students as they entered 10th grade in fall 2013, 
following their ninth-grade Algebra 1 course. Students’ prior year performance was based on the 
2011-2012 K-PREP data, collected in the spring of eighth grade, as well as prior years. Table 1 1 
presents descriptive results for these data, which essentially reflect MDC students’ performance 
prior to and subsequent to their MDC exposure. As presented earlier in Table 4, the prior 
achievement of MDC students was very similar to students statewide both before and after 
matching. 

Table 1 1 


MDC Students ’ Performance Before and After Intervention: ACT PLAN and K-PREP Mathematics Results 


Variable 

n 

Mean 

SD 

Minimum 

Maximum 

ACT PLAN, Fall 2013 






Algebra 

2,690 

7.42 

2.85 

1 

16 

Total 

2,690 

16.40 

3.07 

2 

30 

K-PREP Mathematics, Spring 2012 

2,690 

203.54 

13.48 

109 

242 
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Quasi-Experimental Analysis of MDC Effects 

This section presents the results of our quasi-experimental design examining the impact of 
MDC on ninth-grade Algebra 1 students’ learning. We begin by describing the treated (or study) 
teacher and student samples for the analysis. We then summarize the matching process we used 
to select similar comparison students and to control for the prior effectiveness of teachers and 
schools. Next, we outline the structure and design of the two hierarchical linear models (HLM) 
we employed to estimate the impact of MDC. Finally, we present the results of MDC’s impact 
on student learning, as judged by ACT PLAN scores, using the two modeling approaches. 

Teacher and student sample. As described earlier, our MDC teacher sample includes all 
ninth-grade Algebra 1 teachers in the six target Kentucky school districts who began teaching 
MDC in either 2010-2011 (Phase 1) or 2011-2012 (Phase 2), and who continued implementing 
MDC in 2012-2013. As we will discuss further, where possible our analyses controlled for the 
prior effectiveness of teachers by calculating the value added to their students using assessment 
scores from prior to the start of the MDC initiative (2008-2009 and 2009-2010). Unfortunately, 
these scores were only available for 24 of the 56 teachers in the post-match treatment sample. 

The eligible student sample for the analysis included all students who were (a) enrolled in a 
ninth-grade Algebra 1 class taught by one of the 56 teachers, and (b) for whom prior 
achievement scores were available. As noted in Chapter 2, the 2,690 students eligible for the 
treatment sample were quite similar to the eligible comparison students statewide on both 
demographic and student achievement variables. The eligible treatment sample did have a 
slightly higher proportion of White students, a slightly lower proportion of Black students, and a 
lower proportion of students who were eligible for free or reduced price lunch than did the 
eligible comparison sample of students drawn statewide. 

Selection of comparison students. As was previously noted, treatment students and 
teachers were not randomly selected to participate in the MDC initiative. Because of this, we 
employed the CEM technique to identify comparison students. CEM is a flexible matching 
approach with many favorable properties, and allows the researcher to specify the precise 
conditions under which a comparison student can be matched with an intervention student. For 
categorical variables, such as race/ethnicity or free/reduced price lunch status, this often entails 
exact matching, while for continuous measures, such as prior outcomes and prior teacher 
effectiveness scores, cut-points for matching can be specified. With this approach, precise cut- 
points can be set on the most important prior indicators, such as prior academic achievement, to 
ensure that, where possible, every treatment student is matched with a suitable comparison. 
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Table 12 summarizes the variables used for the matching. Please note that although we 
included indicators for students, teachers, and schools, all matching was completed at the student 
level. Student characteristics in the model included a number of demographic variables (e.g., 
race/ethnicity categories, gender, and free or reduced price lunch eligibility) as well as prior 
mathematics achievement on state assessments. In addition to controlling for these student 
characteristics, our matching methodology selected comparison students whose teachers had 
similar prior effectiveness. The prior effectiveness variable was produced by calculating each 
teacher’s value added to students’ mathematics learning in 2009-2010. Students under teachers 
without prior effectiveness data were matched to comparison students under teachers with 
missing data as well. Finally, a school prior effectiveness variable was calculated using 10th- 
grade mathematics assessment data. These data were used to ensure that the school effectiveness 
variable was independent of the teacher effectiveness variable in the matching model. 

Table 12 


Summary > of Matching Variables 


Indicator type 

Variable 

Student 

Gender 

Student 

White 

Student 

Hispanic 

Student 

Black 

Student 

Asian 

Student 

Special education 

Student 

Free/reduced price lunch eligible 

Student 

Title I 

Student 

Limited English proficient 

Student 

Prior achievement in mathematics 

Teacher 

Availability of teacher prior effectiveness data 

Teacher 

Teacher prior effectiveness 

School 

School prior effectiveness 


The CEM process was successful in finding similar matches for a large majority of the 
eligible MDC students. As can be seen in Table 13, 86% of the treatment students were retained 
in the sample for the QED study. Furthermore, as previously shown in Table 4, the matching 
process was effective in achieving a close balance with regard to prior student scores and 
demographics, as well as the teacher and school effectiveness indicators. 
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Table 13 


Summary’ of Treatment and Comparison Samples 


Sample 

Treatment 

Comparison 

Eligible sample 

2,690 

28,892 

Matched sample 

2,316 

15,325 


Modeling approach. Two separate two-level HLM models were used to conduct the 
matching. Both of these attempt to model students’ dosage under treated and non-treated teachers 
in ninth-grade Algebra 1 courses. In each model, where possible, measures of teacher 
effectiveness on the outcome measure of interest prior to the MDC intervention were estimated 
and used as value-added controls. Student demographic and prior achievement variables, as well 
as teacher and school prior effectiveness were included in the models. Our estimates therefore 
controlled for observables in two ways, at the matching and modeling stages. The models also 
examined potential interactions between MDC treatment and prior school and teacher 
effectiveness as well as student characteristics. These interaction variables were intended to test 
whether MDC had differential effects on student learning depending on the school, teacher, 
and/or individual student’s standing on the given variable. These interaction analyses should be 
considered highly exploratory and results treated as tentative. 

Table 14 summarizes how observations were defined at each level in the two models. In 
Model 1, the Level 1 observations were student course combinations for the year. Depending 
upon the district and/or school, each student could have more than one Algebra 1 course during 
the school year (i.e., teacher changes from one quarter, semester, or trimester to the next), so 
each student could be represented multiple times at this level. To account for this, a weight was 
applied to Level 1 observations so that each student’s total summed to one. Furthermore, one 
teacher was associated with each student/course observation and thus each individual teacher 
was treated as an independent observation at Level 2. 

Table 14 


Observations by Level for Two Hierarchical Linear Models 


Level 

Model 1 

Model 2 

Level 1 

Student/course combinations 

Student 

Level 2 

Teacher 

Teacher 
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In Model 2, each observation at Level 1 represented one student. Level 2 observations 
represented the Algebra 1 teacher to whom that student was assigned. Students who had more 
than one Algebra 1 teacher were randomly assigned to only one of their teachers. Therefore, each 
Level 1 observation was associated with one Level 2 observation. This issue did not present a 
significant problem, as a substantial majority of students during the 2012-2013 school year were 
associated with only one Algebra 1 teacher. Prior teacher and school effectiveness indicators 
were aggregated as cumulative sums to the teacher/combination level at Level 2. 

Each of these models had different advantages. In Model 1, it was not necessary to remove 
any teacher observations. However, the repetition of students at Level 1 was somewhat 
nonstandard and therefore the standard errors may have been underestimated. On the other hand, 
the structure of Model 2 required the elimination of a small amount of infonnation on teacher 
impact. Despite this, the structure of the model was more standard, and therefore provided a 
higher level of confidence regarding the standard errors. Overall, we favored Model 2 over 
Model 1 because we had greater confidence in the standard errors. Because of this, the following 
section will focus on the Model 2 results. Detailed results from both models, which generally 
show a high level of consistency across model specifications, can be found in Appendix E. 

As was previously noted, although prior teacher effectiveness was a variable of interest, it 
was missing for some teachers. We assume that this was because they were relatively new to the 
system or were not teaching ninth grade at the prior time point. To deal with this issue, treatment 
teachers whose prior effectiveness data were missing were matched with comparison students 
whose teachers also had missing scores. In these cases, the teacher effectiveness scores were set 
at zero. Since we had greater confidence in the match for those teachers who were not missing 
data from the period prior to the intervention, we created a “missing” variable to test the effect of 
these missing data: the interaction between the treatment and teachers who were missing their 
prior effectiveness data. Because some power was lost by doing this, we also tested the joint 
significance of the main MDC effect and its interaction with the missing variable. As these 
analyses showed no significant interaction effects for these missing data, the interaction term 
between treatment teachers and teachers who were missing their prior effectiveness data was 
excluded from subsequent analyses. 

HLM results. Results from the Model 2 HLM model on students’ ACT PLAN scores for 
algebra are displayed in Table 15. While the model did control for all of the student, teacher, and 
school indicators previously discussed, we limit our presentation to the intervention effects of 
interest. These effects included potential interactions between the MDC treatment, prior teacher 
and school effectiveness, and student characteristics. A statistically significant interaction effect 
indicates that MDC had differential effects on students’ learning, depending upon their school’s 
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prior effectiveness, teacher’s prior effectiveness, individual demographics, and/or prior 
achievement. These interaction analyses should be considered exploratory and any results should 
be treated as tentative. 


Table 15 

MDC Effects on ACT PLAN Algebra Scores, Including Interaction Effects (Model 2) 


ACT PLAN mathematics (algebra) 

Model 2 Coefficient (SE) 

MDC treatment 

0.130 (0.030)** 

School effectiveness 

0.309 (0.103)** 

Teacher effectiveness 

0.060 ( 0.087) 

MDC treatment by teacher effectiveness interaction 

Treatment by demographic characteristic interactions 

0.420 (0.178)** 

Gender 

0.005 (0.026) 

Special education 

0.070 (0.044) 

Free/reduced price lunch eligible 

0.027 (0.039) 

Prior achievement 

0.030 (0.016) 


Note. Fixed effects for demographic predictors not shown. 

**p< .01. 


A statistically significant positive effect was found for MDC treatment. To provide a 
benchmark for interpreting this effect, we used a relatively new methodology to convert the 
effect size into a gross indicator of the number of months of learning it represents (see Hill, 
Bloom, Black, & Lipsey, 2007). Using this approach, we used available data to estimate the 
growth in mathematics scores from eighth to ninth grade. We then determined the proportion of 
typical growth represented by the observed MDC effect size, that is, the MDC effect size divided 
by the effect size expected between eighth and ninth grade. Finally we used this proportion to 
calculate the number of months of additional growth associated with MDC, relative to a nine- 
month academic year. Relative to typical growth in mathematics, this calculation found that the 
effect size for MDC represented 4.6 months of schooling. The effect is quite dramatic given that 
a typical teacher spent less than a month of class time teaching Challenges. 

The results also show a positive interaction effect for prior teacher effectiveness. This 
positive interaction indicates that students whose teachers were more effective prior to using 
MDC, as measured by their students’ prior state assessment scores in mathematics, benefited 
more from MDC compared to students with lower value-added teachers. We speculate that the 
more effective teachers also had higher pedagogical knowledge in mathematics (see, for 
example, Hill et ah, 2007) that enabled them to implement MDC better. 
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Full results for both Model 1 and Model 2, including the effects at both the student and 
teacher level are presented in Appendix E. The results from these two modeling approaches were 
very similar, although there was one notable difference. In Model 1, the interaction between 
treatment and prior student achievement was significant. In contrast, the interaction between 
treatment and prior teacher effectiveness approached but did not reach the significance threshold. 
These findings were reversed in Model 2, such that the interaction between treatment and prior 
teacher effectiveness was significant, while the interaction between treatment and prior student 
achievement approached but did not reach the significance threshold. The student prior 
achievement interaction would suggest that students entering MDC instruction with relatively 
higher prior performance benefited the most. This matches our observation that the Challenges 
seek to develop and draw on students’ conceptual understanding of mathematics, though this is 
likely more accessible to higher achieving students. 
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Chapter 5: 

Relationship Between MDC Implementation and Student Learning 

This chapter shares exploratory analyses that were conducted to investigate relationships 
between MDC implementation and student learning, based on log, survey, and MDC artifact 
measures. Drawing on the QED analyses described in the prior chapter, our approach used HLM 
to identify implementation variables associated with teacher effectiveness during the MDC study 
year, after taking account of prior teacher effectiveness. Implementation data were available for 
MDC teachers only, so all of the analyses presented in this chapter are within treatment. 

Methodology for Exploratory Analyses 

A first step was constructing our measure of teacher effectiveness. Bayes estimates for each 
teacher’s effectiveness for the study year were saved from HLM Model 1 described in Chapter 4. 
Since the teacher-level variables in this model included effectiveness prior to MDC, our teacher 
effectiveness measure should essentially be considered a measure of the value MDC added to 
effectiveness. 

A second step was to identify implementation variables that appeared related to teacher 
effectiveness. During this stage of the analysis, we explored correlations between log/survey 
responses and teacher effectiveness as well as a cross-tabulation of log and survey responses 
with teachers of high, medium, and low prior effectiveness, with cases equally represented in 
each group. Whether teachers’ participation in MDC was mandatory or voluntary as well as their 
attitudes about the extent and helpfulness of teacher collaboration emerged as potentially 
important. Other variables potentially related to teacher effectiveness included items on the 
assessments and feedback strategies, such as time spent analyzing the pre-assessments and 
providing students with feedback on the pre-assessments and subsequent MDC activities. Since 
both the pre-assessment and feedback strategies are unique to the Challenges, the patterns of 
results seemed to suggest that teachers who more faithfully implemented MDC and its tenets of 
productive struggle were more successful. 

Separate HLM analyses were used to examine the relationship between each potentially 
promising variable and teacher effectiveness. The analysis of log variables included complete 
data for 25 teachers. All Level 1 covariates, as used in the QED model, were group-mean 
centered within this sample and weights normalized. Teacher survey analyses included complete 
data for 30 teachers. 

Finally, variables yielding consistent, statistically significant results for both the correlation 
and preliminary HLM analyses were retained for a combined analysis (i.e., an HLM model 
including all of the identified implementation variables as moderators of teacher outcome year 
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effectiveness). At Level 2, these included the variables listed in Table 16, as well as our prior 
teacher effectiveness variable. 


Table 16 


Statistically Significant Variables From the Exploratory’ Analyses 


Source 

Variable 

Prompt/response option 

Teacher log 

Reviewed pre-assessment 
with class 

How did you review students’ answers to the pre-assessment? 

Option selected: With the class 

Teacher survey 

Special education teacher 

What is/are your current position(s)? Option selected: Special 
Education 

Teacher survey 

Voluntary MDC 
participation 

How would you describe your participation in the MDC initiative? 
Option selected: Voluntary MDC participation 

Teacher survey 

Majority of students 
improved their conceptual 
understanding 

When I taught my most recent MDC Lesson, the majority of my 
students improved their conceptual understanding. Coding: 0 = no, 

1 =yes. 

Teacher survey 

Informal teacher 
collaboration 

About how often do you have informal discussions (as opposed to 
scheduled meetings) with your MDC colleagues to discuss student 
work, instructional strategies, or teaching approaches? Coding: 

1 = at least once a week, 2 = every other week, 3 = once a month, 

4 = once per quarter/trimester/semester, 5 = never. 


Combined Model: Teacher Log and Survey Variables 

The final estimation of fixed effects, with robust standard errors, is displayed in Table 17. 
The results indicate that higher prior student achievement is associated with higher teacher 
effectiveness in the outcome year, and Title I status is associated with lower effectiveness. Of 
direct interest to our study, statistically significant relationships showed that teachers’ voluntary 
participation in the study was associated with higher levels of effectiveness, as were teachers’ 
perceptions of how much MDC improved their students’ conceptual understanding. Lower 
effectiveness was associated with teachers who reviewed the pre-assessment results with their 
class as well as with teachers who taught special education. Although the helpfulness of teacher 
collaboration was positively related to teacher effectiveness, results did not reach statistical 
significance. 
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Table 17 

Final Estimation of Fixed Effects With Robust Standard Errors (2012-2013) 


Fixed effect 

Model coefficient (SE) 

Level 1 variables 

Female 

0.013 (0.036) 

White 

-0.070 (0.078) 

Flispanic 

-0.144 (0.104) 

Black 

-0.048 (0.103) 

Asian 

-0.058 (0.343) 

Limited English proficient 

-0.167 (0.179) 

Special education 

0.048 (0.038) 

Free/reduced price lunch eligible 

0.036 (0.034) 

Title I 

-0.089 (0.033)** 

Prior achievement 

0.486 (0.017)** 

Level 2 variables 

Teacher effectiveness 

0.822 (0.108)** 

Reviewed pre-assessment with class 

-0.873 (0.217)** 

Special education teacher 

-0.742 (0.083)** 

Voluntary MDC participation 

0.316 (0.057)** 

Improved conceptual understanding 2 

-0.496 (0.073)** 

Teacher collaboration 3 

-0.071 (0.042) 


a These items were reverse coded. Because of this, negative coefficients indicate 
an association with increased effectiveness. 

**p< .01. 


Relationship Between Artifact Implementation Measure and Teacher Effectiveness 

As noted earlier, CRESST created indicators of implementation quality based on an 
analysis of the pre- to post-assessment performance for four of the most commonly implemented 
Challenges by study teachers. The indicators were based on students’ pre- to post-assessment 
improvement in content accuracy (i.e., short answer and multiple choice items), quality of 
students’ explanations, and the presence of misconceptions that are common for the targeted 
standards. Although recognizing serious limitations in available data, we then examined the 
relationship between these measures of implementation quality and teacher effectiveness. 

A first step in our analysis was to run an HLM model with the purpose of identifying 
individual MDC teacher’s effectiveness on improving student performance on Classroom 
Challenge assessments. As was previously noted, because different teachers submitted student 
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artifacts for different Challenges, it was important to control for potential differences in the 
Challenges when creating our estimates. Our ratings and analyses were limited to four of the 
Challenges most commonly implemented (i.e., FAL03, FAL13, FAL16, and FAL22). A two- 
level HLM model was used to estimate perfonnance on the post-assessments by controlling for 
the effect of each Challenge as well as pre-assessment scores at both the student and teacher 
levels. Separate models were initially run for each rating (i.e., accuracy, explanation, and 
misconceptions) and then, based on the results, another model was run that combined the mean 
scores for both the accuracy and explanation ratings. Bayes estimates for each teacher’s 
effectiveness at improving student performance on the Challenge assessments for each rating 
were saved. Due to the small sample and the lack of a fully crossed design with respect to 
teachers and student artifacts for the Challenges, these analyses should be considered 
exploratory. 

A second step was to examine the relationship between teacher effectiveness on each of the 
Challenge assessment ratings and teacher effectiveness on state PLAN scores. During this stage 
of the analysis, we explored correlations for the effectiveness scores for the individual Challenge 
ratings as well as for the combined accuracy-explanation score. Table 18 presents the 
correlations between these various effectiveness scores. As can be seen, both the accuracy and 
explanation scores showed strong positive relationships with teacher effectiveness on PLAN, and 
the combined accuracy-explanation score showed a very strong, significant relationship. That is, 
teachers who better implemented the Challenges, as evidenced by their students’ improvement 
on Challenge assessments, also were more effective in having an impact on student scores on the 
PLAN state assessment. These results are suggestive of a relationship between quality of MDC 
implementation and student learning, although given the data limitations already discussed, the 
results should be treated as highly tentative. 

Table 18 


Pearson Correlations for Teacher Effectiveness on Classroom Challenge Ratings and PLAN Scores 



1 

2 

3 

4 

5 

1 . Accuracy and explanation (combined) 

— 





2. Accuracy 

.839** 

— 




3. Explanation 

.775** 

.487 

— 



4. Misconceptions 

.248 

.493 

-.198 

— 


5. PLAN Algebra 

.746** 

.581* 

.527* 

.251 

— 


Note. Aggregate scores for item types control for student pre-assessment scores. 
*p< .05. **/?< .01. 
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Chapter 6: 

Summary and Conclusions 

This report has described CRESST’s study of the implementation and effects of MDC in 
early-implementing ninth-grade Algebra 1 classrooms in Kentucky. Examining how MDC 
supports secondary teachers and students’ transition to the Common Core, the study addresses 
the following evaluation questions: 

1 . How do teachers implement MDC? 

2. What is the impact of MDC on student learning? 

3. What conditions and contexts, including implementation quality, influence MDC 
effectiveness? 

In the following sections, we consider contextual factors that are important in interpreting study 
results before summarizing our findings with regard to each question. We conclude with 
implications and next steps for research and practice. 

Contextual Considerations 

The nature and generalizability of the sample present important limitations for the study. 
First, the study addressed only a subsample of those schools, teachers, and students across the 
country, including in Kentucky, who currently are implementing MDC. The study included only 
teachers and students in those districts and schools that were early MDC implementers, and of 
these, only those in ninth-grade Algebra 1 courses. Because of the Foundation’s interest in a 
rigorous quantitative study, our design required common outcome measures and could not 
accommodate scores from different grade levels or courses. Further, the study focused on 
teachers who had at least one year of prior experience in implementing MDC so that it would not 
be attempting to judge intervention effects as teachers were initially learning how to implement 
the Challenges. Thus, the study was limited to districts that were funded in both Phase 1 and 
Phase 2 of MDC’ s initial pilot and rollout. 

As a result, study power and generalizability were limited. The rigorous quasi-experimental 
design included students taught by 46 ninth-grade Algebra 1 teachers and a carefully matched 
comparison group, which provided limited power to detect moderate program effects. Our ability 
to identify relationships between MDC implementation and outcomes was even more 
constrained, as only about two thirds of the teachers agreed to participate in the implementation 
components of the study. 

The representativeness of the study sample further limited the generalizability of any study 
findings. Demographically and in prior achievement, our MDC sample generally looked similar 
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to the state as a whole. However, the study could not control for unobserved variables that may 
have influenced student success and indeed, by virtue of their willingness to participate in early 
MDC piloting, the districts and schools in the study may well be at least somewhat atypical. 

That study teachers had minimum prior experience implementing the intervention was still 
another important contextual consideration. Before the study year, the majority of these teachers 
had only one year of prior experience implementing MDC and, in fact, that prior year included 
both initial learning and implementation. One year was hardly adequate time for teachers to 
meaningfully integrate and become effective with new practices — and the MDC student- 
centered, formative assessment lessons certainly involved transformation of usual teacher- 
centered mathematical practices. 

Intervention dosage was still another factor worth consideration. For the study year, most 
teachers implemented approximately six Classroom Challenges, which translates into only about 
12 to 18 days of instruction over the course of the year. It is ambitious, in short, to expect MDC 
to have a measurable impact on student learning at this early point in implementation. 

How Did Teachers Implement MDC? 

End-of-year survey responses indicated that the majority of teachers participated in 
professional development to support their MDC implementation and felt well supported by 
district leadership and their teacher colleagues. School leadership support for implementation 
appeared more variable across teachers and schools. 

Both log and survey results suggested that the great majority of teachers mostly or fully 
adhered to MDC guidelines in implementing individual Challenges, but varied substantially in 
the specific timing of and strategies used during the various components. Teachers reported 
frequent use of formative assessment, using a variety of strategies to monitor their students’ 
ongoing learning and taking action consistent with MDC’s productive struggle philosophy when 
misunderstandings and/or problems were observed, again with substantial variation in the 
strategies used. However, analysis of students’ pre- to post-assessment responses on the most 
commonly implemented Challenges indicated that teachers had difficulty in helping students 
achieve fully the Challenge goals. 

Such difficulties were to be expected at this early stage of implementation, particularly as 
the Challenges demanded sophisticated pedagogical and assessment strategies. The variation in 
reported strategy, in fact, may be an indicator that teachers had not yet solidified their MDC 
practice, but instead were trialing a variety of potential actions. The inconsistency in strategy 
may provide one reason why we were unable to find strong relationships between any single 
MDC implementation measure and student learning outcomes. Further, the quality and 
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effectiveness of implementation depended on how well teachers were able to implement 
Challenge activities and respond to students’ needs in classroom interaction, among other 
unobservables. Challenges not implemented with effective teaching, assessment, and learning 
strategies would not be expected to have a strong effect on student learning. 

How Did MDC Affect Student Learning? 

Teacher perspectives. Survey responses showed that teachers were very positive about 
key MDC pedagogical strategies and felt that the Challenges benefited their students’ conceptual 
understanding and mathematical thinking. At the same time, they reported that sizable 
proportions of their students struggled during the Challenges and only about half achieved 
intended goals. As with the analysis of students’ pre- and post-assessments, these results suggest 
that participating teachers may have needed additional help with the implementation of these 
Challenges and with their MDC instruction. 

CRESST mathematics assessment. Performance on the CRESST mathematics 
assessment underscores teachers’ reports that many students struggled with the Challenges. 
Because the specific Challenges that were implemented varied across schools and/or teachers, 
the mathematics assessment focused on the standards and content addressed by the five that were 
most frequently used. This design constraint meant that not all students in the study had an equal 
opportunity to learn all of the assessment content and, in fact, based on the OTL survey 
completed by teachers at the end of the study year, it seems likely that relatively few classes 
engaged in two or more of the five Challenges assessed. Even with this caveat, student 
performance was very low, suggesting the limits of the conceptual understanding students 
achieved. Student motivation may have been another mitigating factor as the test was 
administered at the end of the school year. Nonetheless, the results suggest the challenge of 
moving student performance to the expectations of the Common Core. 

MDC impact on student learning. Our QED methodology used CEM to identify a group 
of comparison students who were demographically and academically similar to the study’s MDC 
students. The matching was done at the student level, but accounted not only for student 
demographics, prior achievement, and course enrollment, but also for the prior effectiveness of 
teachers and schools. The resulting treatment and comparison student samples were used to test 
the effects of MDC on students’ performance on the ACT PLAN, which is administered in the 
fall to 10th- grade students as part of Kentucky’s state assessment program. The analysis used 
two separate, two-level HLM models to test MDC effects, each modeling students’ dosage under 
treated and non-treated teachers in ninth-grade Algebra 1 courses, and each incorporating 
measures of teacher effectiveness as additional value-added controls. Student demographic and 
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prior achievement variables, as well as school prior effectiveness, were also included in the 
models, as were the interactions of these variables with the MDC treatment. Our estimates 
therefore controlled for observables in two ways, at the matching and modeling stages. 

Results were consistent for both models, suggesting the robustness of these findings. MDC 
showed a positive, statistically significant effect on students’ ACT PLAN performance in 
algebra, indicating that students who experienced MDC learned more than students who did not 
have the benefit of MDC. The observed effect size for MDC represented 4.6 months of schooling 
(see Hill et ah, 2007), decidedly positive news for the intervention. 

What Conditions and Contexts Influenced MDC Effectiveness? 

Interesting interaction effects emerged from our QED analysis that point to conditions and 
contexts that influenced MDC implementation and impact. In particular, results were suggestive 
of a positive interaction between students’ prior achievement and/or teachers’ effectiveness prior 
to MDC implementation. The two statistical models utilized in study analyses revealed similar 
directions in interaction findings but differences in statistical significance. That is, in one model, 
students’ prior achievement showed a statistically significant interaction with the treatment, and 
prior teacher effectiveness showed a positive interaction that approached statistical significance; 
the statistically significant findings for these interactions were reversed in the second model. 
Given the likely relationship between teacher quality and student achievement, these 
inconsistencies are understandable. The combined findings suggest MDC students who were 
relatively higher achieving prior to their MDC experience and/or those whose teachers were 
more effective prior to using MDC benefited more from the intervention compared to their peers. 

Conclusions 

In summary, MDC showed promising, positive results in supporting teachers’ transition to 
the Common Core expectations for college and career readiness and in improving student 
learning. At the same time, however, study findings suggest challenges that MDC will need to 
overcome to move to higher levels of success. Next, we summarize our perspective on major 
study implications. 

Positive effects on student learning. Study findings of MDC’s statistically significant, 
positive effects on Kentucky students’ ACT PLAN perfonnance are worth underscoring, 
particularly in light of both study teachers’ limited prior experience implementing the tools and 
the limited dosage students experienced. As previously noted, study teachers had only one or two 
years of experience with MDC prior to the study year, and the great majority had only one year 
of prior experience. Research on teachers’ implementation of new practices suggests that this is 
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insufficient time for teachers to become fully comfortable and competent with the kinds of new 
pedagogical practices that MDC represents (Coburn, 2003; Hargreaves & Fullan, 2012). 

Student intervention dosage is another important consideration. In general, the longer and 
more intensive the treatment, the more likely an intervention is to show measurable effects. Most 
teachers implemented MDC for only 12-18 days of instruction, a small fraction of the full 
academic year in which the study took place. Nonetheless, the study found a statistically 
significant learning effect equivalent to approximately 4.6 months of schooling. 

Positive effects on teachers. The effect found for student learning was matched by teacher 
enthusiasm for MDC. Study teachers were positive about the professional development they 
received and reported that they found the tools helpful and effective in meeting a variety of 
goals, including implementing the new standards, using formative assessment, incorporating 
more complex thinking and problem solving into curriculum and instruction, and improving 
student learning. Teachers’ reports about their fidelity of tool implementation provided 
additional evidence of their positive attitudes. 

Struggles in moving to higher standards. While our study found positive effects on 
teachers and students, findings also demonstrated the challenge of moving to standards that are 
more rigorous. We see evidence of this challenge in students’ low performance on measures 
specifically designed to reflect the deeper learning demands of the new CCRS, and in teachers’ 
reports that sizable proportions of their students struggled with the Classroom Challenges and 
did not achieve success with Challenge goals. Our analysis of MDC classroom artifacts also 
suggested that teachers and students struggled in their implementation efforts, as would be 
expected given this early stage of implementation. 

That some teachers and students struggled is not meant to imply that the new standards are 
unattainable or that CCRS expectations for students should be reduced. After all, we know that 
returning to prior standards will not get our children to 21st century success. However, the 
evidence does suggest that change will not come overnight and that both teachers and students 
will need support to meet the challenge. The issue is twofold: (a) how to address the needs and 
better prepare students and teachers who may not yet be ready to be successful with the 
challenges of MDC; and (b) how to modify and/or adapt MDC tools to scaffold teacher and 
student learning more effectively. 

Achievement gap implications. Although we regard findings of interactions between 
MDC and both student ability and teacher prior effectiveness as tentative and subject to further 
validation, they raise important questions for policy and practice. The overall results indicated 
that MDC was effective for all study students, but the interaction findings indicated that initially 
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higher achieving students and/or those taught by initially more effective teachers benefited more 
than did their comparable peers. Such a finding makes intuitive sense in that lower achieving 
students have most likely been exposed to the “drill and kill” test preparation curriculum of the 
past, are least likely to have acquired the prior grade knowledge and skills, and are also the least 
likely to have engaged in the mathematical practices expected by the new standards. Similarly, 
the possible interaction between prior teacher effectiveness and MDC tells a similar story. 
Students of teachers who initially were relatively more effective appeared to benefit more from 
MDC than did students of initially relatively less effective teachers. 

Strengthening implementation. Although teachers reported implementing all components 
of MDC, the findings suggest substantial variation in how teachers implemented each component 
and in the relative time and specific strategies they used in doing so. The study did not achieve 
strong findings with regard to what aspects of implementation mattered most or what specific 
strategies were most effective. The findings are suggestive, however, of some factors that might 
be important to success: District support for MDC was clear across the sample, yet principal or 
local school support was more variable, suggesting a potential problem point. Teachers found 
their peers highly collaborative and helpful in supporting MDC implementation. The extent of 
such collaboration tended to be associated with MDC success. Moreover, having a teacher who 
volunteered to participate in MDC, rather than having participation required, was associated with 
positive effects on student learning. 

Concluding thoughts. In summary, our study reveals that study teachers were enthusiastic 
about MDC, and that MDC showed important effects on student learning. Even so, study results 
suggested areas for improvement. Mathematics teachers who have deep content-pedagogical 
knowledge in their course content are likely to be more successful in implementing MDC, and 
many teachers may need help to reach these deeper levels of knowledge. Additional supports for 
struggling students, and training on how to implement MDC successfully with diverse and low- 
achieving students could potentially help close achievement gaps. 

We leave it to future research to examine the generalizability of these findings in the larger 
samples of teachers and schools that are now implementing MDC. Cost-effectiveness studies 
should be of interest. Future research and development also should continue the quest to identify 
the most critical aspects of implementation in improving student learning and the key 
infrastructure and supports that students and teachers who currently are struggling need to propel 
their success. 
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Appendix A: MDC Instruments and Rubrics 
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Exhibit Al: MDC Teacher Log 


Thank you for taking the time to complete your teacher log for MDC instruction. This log should 
take no longer than five (5) minutes to complete. As you answer the questions below be sure to 
limit your responses to (a) the classroom challenge (FAL) you just completed; (b) the class 
period listed in your reminder email (in the event that you teach MDC to several classes). 


If you have already completed the log and would like to upload student work, click "upload 
student work" below. Otherwise, please click on "Continue with log." 

O Skip log to upload student work 
O Continue with log 


Classroom Challenge (FAL) title or number (e.g., “Boomerangs”)” 


How many days did you spend teaching a Classroom Challenge (FAL) this week? 

1 2 3 4 5 

o o o o o 


What was your purpose for teaching this particular Classroom Challenge? (check all that apply) 

□ District mandate 

□ Review for test 

□ Introduce new content 

□ Extending/reinforcing new content 

□ Practice problem solving 

□ Other, please specify 
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How many days of NON-Classroom Challenge instruction did you devote to this particular 
mathematical content prior to teaching this Classroom Challenge? 


When was the last day of NON-Classroom Challenge instruction on this particular mathematical 
content? 

O Earlier this week 
O Last week 

O Within the past few weeks 
O Over a month ago 


To what extent did you follow the detailed Classroom Challenge lesson plan instruction? 


Not at all 

Somewhat 

Mostly 

Completely 

o 

o 

o 

o 


Which of the following activities did you do as part of Classroom Challenge instruction? (check 
all that apply) 

□ Pre-assessment 

□ Small-group collaborative work 

□ Whole-class plenary 

□ Post-assessment 


If you did not assign a post-assessment, which of the following best characterizes your reasoning 
(check all that apply) 

□ I ran out of class time 

□ I plan to assign it tomorrow 

□ It seemed redundant 

□ My students understood the material 

□ It was too difficult 

□ Other, please specify 
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Consider the total amount of class time spent on Classroom Challenge instruction. What 
proportion of time was spent on the following classroom activities? (Total should add up to 
100 %) 


Pre-assessment 

Small-group collaborative work 

Whole-class discussion 

Individual work 

Student presentations 

Teacher-led review of content/ 
concepts 

Post-assessment 

Other 

Total: 



Pre- Assessment 

When did students complete the pre-assessment? 

O In class at the start of Classroom Challenge 
O Other 


How much time did you spend reviewing students’ answers to the pre-assessment? 

0 min 1-5 min 6-15 min 16-30 min 3 1-60 min >1 hour 

o o o o o o 


How did you review students’ answers to the pre-assessment? 
O Alone 
O With a colleague 
O With the class 
O Other 
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Which of the following characterizes the feedback you gave students on their pre-assessment? 


□ No feedback provided because Classroom Challenge instruction addressed misconceptions 

□ Commented on papers 

□ Graded papers 

□ Wrote questions on papers 

□ Reviewed common errors at start of class 

□ No feedback provided because of time constraints 


To what extent do you agree or disagree with the following statements? 



Strongly 

disagree 

Disagree 

Agree 

Strongly 

agree 

Before starting the Classroom Challenge, 1 had a 
very strong sense of students' misconceptions on this 
topic. 

o 

o 

O 

o 

My understanding of students' misconceptions comes 
from studying their answers on the pre-assessment. 

o 

o 

o 

o 

It was difficult for me to identify students' 
misconceptions from the pre-assessment. 

o 

o 

o 

o 

I relied primarily on the lesson's suggested 
questions/comments when responding to pre- 
assessments. 

o 

o 

o 

o 


Small-group Collaborative Work 

When you heard students struggle in small groups, what did you do? (check all that apply) 

□ Listened but did not intervene 

□ Offered hints and suggestions 

□ Raised questions 

□ Provided the answer 

□ Asked another student to explain concept 

□ Stopped class and reviewed concept 
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What percentage of students struggled to understand the underlying mathematical concept? 


<5% 

5-25% 

25-50% 

50-75% 

75-99% 

100% 

Not sure 

O 

O 

o 

O 

O 

o 

o 


Whole-class Plenary 

For each of the types of questioning described below, please indicate whether or not it was 
incorporated into today’s lesson. Each type is accompanied by several examples to illustrate 
what kinds of questions are included in that category. 



Yes 

No 

IRE (Initiate-Respond-Evaluate): (Yes/No questions, questions that ask 
students to recall facts or definitions) 

O 

o 

Self-Reflection: (Why do you think that? Why is that true? How did you 
reach that conclusion?) 

o 

o 

Reflection-on-Others: (Do you agree? Does anyone have a different way to 
explain it? Would you ask the rest of the class that question?) 

o 

o 

Pattern-Finding and Conjecturing: (How did you predict the next case? 

What is similar and what is different about your solution and his/hers? Do 
you see a pattern?) 

o 

o 

Mathematical Reasoning: (Does that always work? Is that true for all cases? 

Can you think of a counter example?) 

o 

o 


What percentage of students presented their solutions to the whole class? 


<5% 

5-25% 

25-50% 

50-75% 

75-99% 

100% 

Not sure 

o 

O 

o 

O 

O 

o 

o 


What percentage of students articulated the reasoning behind their solutions to the whole class? 


<5% 

O 


5-25% 

O 


25-50% 

O 


50-75% 

O 


75-99% 

O 


100 % 

o 


Not sure 

o 
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Post Assessment 


When did students complete the post-assessment? 
O In class at the end of the Classroom Challenge 
O Homework 

O Other, please specify 


What percent of your students do you think gained a strong grasp of the material by the end of 
the lesson? 


<5% 

O 


5-25% 

O 


25-50% 

o 


50-75% 

O 


75-99% 

O 


100 % 

o 


Not sure 

o 


Please indicate the approximate date of your next Classroom Challenge. We recognize that this 
may change, (mm/dd/yyyy) 


Please scan and upload a full class set of student work on the Classroom Challenge. We would 
like both the Classroom Challenge pre-assessments and post-assessments. You can attach PDFs 
or Word files. If you would prefer to send us hard copies, please email Debbie La Torre 
Matrundola at latorre@cse.ucla.edu. 

If you would like to upload the student work later, you can reenter the log and click "Skip Log to 
Upload Files" on the first page. 


Full class set of Classroom Challenge pre-assessments 


Browse 


Full class set of Classroom Challenge post-assessments 


Browse 
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Additional student work (optional) 


Browse 


ADDITIONAL COMMENTS: 

We are very interested in your feedback. Please let us know if you have any questions or 
concerns about this log. Thank you! 


52 





Exhibit A2: Artifact Exemplars and Scoring Rubrics 


FAL 03, Solving Linear Equations in Two Variables 
Scoring Exemplar 


SoKnnfl LmMr Equations m Two Vanabte s Student MjteruH 

I Notebooks and Pens 


BdiVmon 


A store sells pens at S2 and notebooks at S?> 
n - number of notebooks sold 
p - number of pens sold 
The following equations are true 
4 n=p 

bn * 2p =39 

Here is wfiat Dan and Emma thnk the equations mean 


I think the tccood caution 

mcanH ih.ii l he uurc hiU 
5 notebook, jnd 2 pen,. 


t think the firu cqualum 
mean, that the wore telh 
four lime, a, man, 
notebook, a, pen,. 



Dan 


Emma 



Are Dan and Emma correct? 

No. they are both incorrect 


If you think Dan is wrong, explain the mistake and explain what you think the equation means. 

Don is wrong becouse he thought the equation meant there were 4 notebookssold for every pen. bethought n 
stood for notebook rather then the number of notebooks. The equation actually neons if you multiply the 
number of notebookssold by 4that equalsthe number of pens said. 


If you think Emma is wrong, ex plan the mistake and explain what you think the equation means 


Emma mode the some mistake She also thouglrt that n stood for notebook and that p stood for pen Sndoes 
not mean 5 notebooks - it means 5 multiplied by the number of notebookssold We were told that notebooks 
cost $5 and pens $2 so 5n gives you the amount of money earned from selling notebooks, and 2p the amount of 
money earned from selling pens So5n« 2p = 39 means that $39 was made selling pens and notebooks. 

Figure out for yourself the number of pens and the number of notebooks sold in the store 

If 4n is substituted forp m the second equation, you get 5n» 2(4n) = 39 

5 n* Bn: 39 


13m 39 

Divide by 13' n = 3 4 

>ut this in the f rst equation and you get 4 x 3 = p 

And so: p = 12 4 


O 201 1 MARS Urvwfvty of Nottingham S-1 
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FAL 03, Solving Linear Equations in Two Variables 
Quality Rubric 


Item: Notebooks and Pens-01 
Prompt: Are Dan and Emma correct? 


[unscored] 


Item: Notebooks and Pens-02 

Prompt: If you think Dan is wrong, explain the mistake and explain what vow think the equation 
means. 


Accuracy/Precision 

Explanation (Dan’s mistake) 

Explanation (interpretation of 
equation) 

• (1) The student has 
concluded that Dan is 
wrong. 

• (0) The student has 
concluded that Dan is 
right, or leaves blank. 

• (3) The student explicitly 
describes Dan’s 
misinterpretation (e.g. “He 
thought n stood for notebook 
rather than number of 
notebooks ”). 

• (2) The student mentions that 
Dan has misinterpreted n, but 
the explanation lacks 
specificity or clarity. 

• (1) The explanation 
demonstrates no evidence of 
understanding of Dan’s 
misinterpretation. 

• (0) The student does not 
explain Dan’s mistake. 

• (3) The student correctly 
interprets the equation (e.g. 

“If you multiply the number 
of notebooks sold by 4 that 
equals the number of pens 
sold”) 

• (2) The student’s explanation 
shows some evidence of a 
correct interpretation of the 
equation (e.g. the 
explanation is partially 
correct or uses ambiguous or 
confusing language). 

• (1) The interpretation 
demonstrates no evidence of 
understanding. 

• (0) The student does not 
interpret the equation. 

Accuracy/ 

Precision 

Score 


Explanation 

Score 


Explanation 

Score 
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Item: Notebooks and Pens-03 

Prompt: If you think Emma is wrong, explain the mistake and explain what vow think the 
equation means. 


Accuracy/Precision 

Explanation (Emma’s mistake) 

Explanation (interpretation of 
equation) 

• (1) The student has 
concluded that Emma 
is wrong. 

• (0) The student has 
concluded that Emma 
is right, or leaves 
blank. 

• (3) The student explicitly 
describes Emma’s 
misinterpretation (e.g. “She 
thought that n stood for 
notebook and that p stood for 
pen”). 

• (2) The student mentions that 
Emma has misinterpreted n 
and p, but the explanation 
lacks specificity or clarity. 

• (1) The explanation 
demonstrates no evidence of 
understanding of Emma’s 
misinterpretation. 

• (0) The student does not 
explain Emma’s mistake. 

• (3) The student correctly 
interprets the equation (e.g. 
“$39 was made selling pens 
and notebooks”). 

• (2) The student’s explanation 
shows some evidence of a 
correct interpretation of the 
equation (e.g. the 
explanation is partially 
correct or uses ambiguous or 
confusing language). 

• (1) The interpretation 
demonstrates no evidence of 
understanding. 

• (0) The student does not 
interpret the equation. 

Accuracy/ 

Precision 

Score 


Explanation 

Score 


Explanation 

Score 



Item: Notebooks and Pens-04 

Prompt: Figure out for yourself the number of pens and the number of notebooks sold in the 
store. 

Accuracy/Precision 

• (2) The solution is correct (e.g. “3 
notebooks and 12 pens were sold”). 

• (1) There is evidence of a viable solution 
strategy (e.g. the student has attempted to 
solve with substitution, but has made an 
algebra or arithmetic error). 

• (0) There is no evidence of a viable 
solution strategy or no solution was 
written. 

Accuracy/Precision Score 
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FAL 03, Solving Linear Equations in Two Variables 
Common Issues (Misconceptions) Rubric 


Item: Notebooks and Pens-01 
Prompt: Are Dan and Emma correct? 

[unscored] 


Item: Notebooks and Pens-02 

Prompt: If you think Dan is wrong, explain the mistake and explain what vow think the equation 
means. 


Code 

Common Issue 

Example of Present 
Common Issue 

Is the Common 
Issue present? 
(Yes/No) 

Variable 

Misrepresentation 

Student assumes that the 
letter stands for an 
object not a number. 

Student says that the 
statements are correct. 



Item: Notebooks and Pens-03 

Prompt: If you think Emma is wrong, explain the mistake and explain what vow think the 
equation means. 


Code 

Common Issue 

Example of Present 
Common Issue 

Is the Common 
Issue present? 
(Yes/No) 

Variable 

Misrepresentation 

Student assumes that the 
letter stands for an 
object not a number. 

Student says that the 
statements are correct. 
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Item: Notebooks and Pens-04 

Prompt: Figure out for yourself the number of pens and the number of notebooks sold in the 
store. 


Code 

Common Issue 

Example of Present 
Common Issue 

Is the Common 
Issue present? 
(Yes/No) 

Uses Single 

Equation 

Student only uses one 
equation. 

Student finds a value or 
values for n and p that fits 
one equation but not the 
other, such as n = 1 and p = 

4 for the first equation. 


Guess and Check 

Student produces 
unsystematic guess and 
check work. 

Student works out three or 
four seemingly 
unconnected combinations 
of values for n and p. 


Algebraic Mistake 

Student makes algebraic 
mistakes. 

Student makes a mistake 
when manipulating the 
algebra in the equations. 
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FAL 13, Sorting Equations and Identities 
Scoring Exemplar 


Sonne Equaaons and Identities Student Materials Beta V»wn 

Equations and Identities 

1 . Write down an example of an equation that has: 

(a) One solution 

(b) Two solutions 

(c) An infinite number of solutions 

(d) No solutions 


3x = 27 
xi-6= 10 
5x * 45 = 5(x ♦ 9) 
-8 ♦ 2x = 2x 


2 For each of the following statements, indicate whether it is 'Always true.' 'Never true.' or 

'Sometimes true ' Circle the correct answer If you choose 'Sometimes true' then state on the line 
below when it is true The first one is done for you as an example 


x + 2-3 


Always true Never true 

It is true when x = 1. 


Sometimes true 


X-12-X+30 Always true (^Nevertrue 

It is true when 


Sometimes true 


2(x + 6)-2x+ 12 (Always true 
It is true when 


Never true 


Sometimes true 


3(x- 2)-3x-2 Always true 
It is true when 


Sometimes true 


Cx + 4)--r + 4' 


Always true 
It is true when 


Never true 

x = 0 


■times i 


x s +4-0 Always true (^Never true^) 


Sometimes true 


It is true when 


Which of the equations in question 2 are also identities' 7 

2(x ♦ 6) = 2x ♦ 12 


In your own words, explain what is meant by an identity 

An identity is used to descrke on equation that is akvoys true. An example is 3(x»4) = 3xel2. This 
equation has an infinite number of solutions and for any value of x will always be true. 


O 201 1 MARS Unrv*rvty of Nottingham 
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FAL 13, Sorting Equations and Identities 
Quality Rubric 


Item: Equations and Identities-0 1 

Prompt: Write down an example of an equation that has: 

(a) One solution. 

Accuracy/Precision 

• (1) The student’s response is correct (i.e. an 
equation with one solution). 

• (0) The student’s response is incorrect (e.g. an 
equation with no or multiple solutions, an 
expression instead of an equation). 

Accuracy/Precision Score 


(b) Two solutions. 

Accuracy/Precision 

• (1) The student’s response is correct (i.e. an 
equation with two solutions). 

• (0) The student’s response is incorrect (e.g. an 
equation with fewer or more than 2 solutions, 
an expression instead of an equation). 


Accuracy/Precision Score 


(c) An infinite number of solutions. 

Accuracy/Precision 

• (1) The student’s response is correct. 

• (0) The student’s response is incorrect (e.g. an 
equation with a finite number of solutions, an 
expression instead of an equation). 


Accuracy/Precision Score 


(d) No solutions. 


Accuracy/Precision 

• (1) The student’s response is correct (i.e. a 
equation with no solutions). 

• (0) The student’s response is incorrect (e.g 
equation with one or more solutions, an 
expression instead of an equation). 

Accuracy/Precision Score 

an 
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Item: Equations and Identities-02 

Prompt: For each of the following statements, indicate whether it is ‘Always true,’ ‘Never true,’ 
or ‘Sometimes true.’ Circle the correct answer. If you choose ‘Sometimes true’ then state on the 
line below when it is true. The first one is done for you as an example. 

(i) x + 2 = 3 

[unscored-example item] 


(ii) x - 12 = x + 30 


Accuracy/Precision 

• (1) The student has circled ‘Never true.’ 

• (0) The student has circled a different respc 
or has left the item blank. 

Accuracy/Precision Score 

rnse 


(iii) 2(x + 6) = 2x + 12 


Accuracy/Precision 

• (1) The student has circled ‘Always true.’ 

• (0) The student has circled a different respc 
or has left the item blank. 

Accuracy/Precision Score 

mse 


(iv) 3(x - 2) = 3x - 2 


Accuracy/Precision 

• (1) The student has circled ‘Never true.’ 

• (0) The student has circled a different choic 
has left the item blank. 

Accuracy/Precision Score 

CD 

O 

i-i 


(v) (x + 4) 2 = x 2 + 4 2 

Accuracy/Precision 

• (2) The student has circled ‘Sometimes true’ 
and written the solution x = 0. 

• (1) The student has circled ‘Sometimes true’ 
but has not provided the solution x = 0 or 

• (1) The student has not circled ‘Sometimes 
true’ but has provided the solution x = 0. 

• (0) The student has circled a different choice or 
has left the item blank. 

Accuracy/Precision Score 
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(vi) x 2 + 4 = 0 


Accuracy/Precision 

• (1) The student has circled ‘Never true.’ 

• (0) The student has circled a different choic 
has left the item blank. 

Accuracy/Precision Score 

CD 

O 


Item: Equations and Identities-03(i) 

Prompt: Which of the equations in question 2 are also identities? 


Accuracy/Precision 

• (2) The student has listed only 2(x + 6) = 2x + 
12 . 

• (1) The student has listed 2(x + 6) = 2x + 12 
and one additional equation. 

• (0) The student has not listed 2(x + 6) = 2x + 
12 or has listed more than one incorrect 
equation. 

Accuracy/Precision Score 


Item: Equations and Identities-03(ii) 

Prompt: In your own words, explain what is meant by an identity. 


Explanation 

• (3) The student describes identities as equations 
that are always true and provides an example. 

• (2) The student response indicates correct 
understanding of identities, but lacks precision 
or completeness (e.g. “they are always true”). 

• (1) The student response provides no evidence 
of correct understanding of identities. 

• (0) The student gives no explanation. 

Explanation Score 
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FAL 13, Sorting Equations and Identities 
Common Issues (Misconceptions) Rubric 


Item: Equations and Identities-0 1 

Prompt: Write down an example of an equation that has. . . 


Code 

Common Issue 

Example of Present Common 
Issue 

Is the Common 
Issue present? 
(Yes/No) 

Variable 

Missing 

Student fails to include a 
variable in the equation 

Student writes 5+5=10 as an 
example of an equation with one 
solution in part (a) 

or 

Student writes 5+6=10 as an 
example of an equation with no 
solutions in part (d). 


Expression 
vs. Equation 

Student thinks 
expressions and 
equations are the same 
thing. 

Student writes y+3 for an 
equation with an infinite number 
of solutions in part (c). 


Non-Integer 

Solutions 

Student provides a 
quadratic with non- 
integer solutions as an 
example of an equation 
with no solutions. 

Student writes x 2 + 8x + 13 = 0 
as an example of an equation 
with no solutions in part (d). 
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Item: Equations and Identities-02 

Prompt: For each of the following statements, indicate whether it is ‘Always true,’ ‘Never true,’ 
or ‘Sometimes true.’ Circle the correct answer. If you choose ‘Sometimes true’ then state on the 
line below when it is true. The first one is done for you as an example. 


Code 

Common Issue 

Example of Present Common 
Issue 

Is the Common 
Issue present? 
(Yes/No) 

Distributive 

Property 

Student assumes that 3(x 
- 2) is the same as 3x - 2. 

Student classifies 3(x - 2) = 3x - 
2 as “Always true.” 


Exponent 

over 

Addition 

Student assumes that (x + 
4) 2 is the same as x 2 + 4" 

Student classifies (x + 4) 2 = x 2 + 
4 2 as “Always true.” 


Squaring a 
Negative 

Student assumes that - 
2 2 
(x ) is the same as (-x) . 

Student classifies x +4 = 0 as 
“Sometimes true” and provides 
the case, x = -2. 



Item: Equations and Identities-03(i) 

Prompt: Which of the equations in question 2 are also identities? 

[unscored] 

Item: Equations and Identities-03(ii) 

Prompt: In your own words, explain what is meant by an identity. 

[unscored] 
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FAL 16, Interpreting Algebraic Expressions 
Scoring Exemplar 




Inttrpretng Algebnic Expressons Student Materials Beta Version 

Interpreting Expressions 

1 . Wnte algebraic expressions for each of the following: 

a. Multiply n by 5 then add 4. 

b. Add 4 to n then multiply your answer by 5 

(n ♦ 4) / 5 

c. Add 4 to n then divide your answer by 5. 

3n 2 

d. Multiply n by n then multiply your answer by 3. 

(2nV 

e. Multiply n by 3 then square your answer. 


2. Imagine you are a teacher. Decide whether the following wo* is correct or incorrect 
If you see an error. 

a. Cross it out and replace it with a correct answer. 


b. Explain the error using words or diagrams. 


+ 3 ) - + j >< 


You did not distribute the 2 correctly. 
2(n ♦ 3) is the same as 2n ♦ 2 x 3 or 
2n ♦& 


10 **. - 5 

2 **. - 1 

5 


Correct. 


25 

( 5 m ,) 2 


You did not square the 5. (5n)£ is the same as 
5n x 5n which is 25n 2 . 


When you expand (n ♦ 3)5 you get (n ♦ 3) x (n ♦ 3) 
which is n 2 ♦ 3n ♦ 3n ♦ 9 or n 2 ♦ 6n ♦ 9. (The 
student may draw a diagram.) n 3 

n 


02011 MARS University of Notfcngham 3 


n 2 + 6n + 9 

(**, + s) a - 
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FAL 16, Interpreting Algebraic Expressions 
Quality Rubric 


Item: Interpreting Expressions-0 1 

Prompt: Write algebraic expressions for each of the following: 

(a) Multiply n by 5 then add 4. 


Accuracy/Precision 

• (1) The student’s response is correct (i 
5n+4 or an equivalent form). 

• (0) The student’s response is incorrect. 

Accuracy/Precision Score 

e. 


(b) Add 4 to n then multiply your answer by 5. 


Accuracy/Precision 

• (1) The student’s response is correct (i 
5(n+4) or an equivalent form). 

• (0) The student’s response is incorrect. 

Accuracy/Precision Score 

e. 


(c) Add 4 to n then divide your answer by 5. 


Accuracy/Precision 

• (1) The student’s response is correct (i 
(n+ 4)/5 or an equivalent form). 

• (0) The student’s response is incorrect. 

Accuracy/Precision Score 

e. 


(d) Multiply n by n then multiply your answer by 3. 


Accuracy/Precision 

• (1) The student’s response is correct (i 
3/7 or an equivalent form). 

• (0) The student’s response is incorrect. 

Accuracy/Precision Score 

e. 
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(e) Multiply n by 3 then square your answer. 


Accuracy/Precision 

• (1) The student’s response is correct (i 
(3n) 2 or an equivalent fonn). 

• (0) The student’s response is incorrect. 

Accuracy/Precision Score 

e. 


Item: Interpreting Expressions-02 1 

Prompt: Imagine you are a teacher. Decide whether the following work is correct or incorrect. 
If you see an error: (a) Cross out the expression on the right and replace it with the expression 
that is equivalent to the one on the left, (b) Explain the error using words or diagrams. 

(i) 2(n + 3) = 2n + 3 


Accuracy/Precision 

• (1) The student corrects the error by 
replacing 3 on the RHS with 6. 

• (0) The student does not correct the error. 

Accuracy/Precision Score 


Explanation 

• (3) The student’s explanation 
displays understanding of the 
distributive property (e.g. “You did 
not distribute the 2 correctly”). 

• (2) The student’s explanation 
displays limited understanding of the 
distributive property. 

• The student’s explanation displays 
no understanding of the distributive 
property. 

• No explanation is provided. 

Explanation Score 


10n-5 „ . 

(li) — - — = 2n — 1 

Accuracy/Precision 

• (1) The student concludes the work is 
correct. 

• (0) The student concludes the work is 
incorrect. 


Accuracy/Precision Score 


1 In completing part (b), students may include area diagrams, correct algebraic steps, or other supports for their 

explanation. However, this grading rubric only considers their written explanation. 
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(iii) 


(5 n) 2 - 5 n 2 


Accuracy/Precision 

• (1) The student corrects the error 1 
replacing 5 on the RHS with 25. 

• (0) The student does not correct tt 
error. 

Accuracy/Precision Score 

W 

Le 


Explanation 

• (3) The student’s explanation displays 
understanding of the distribution of 
exponents over multiplication. 

• (2) The student’s explanation displays 
limited understanding of the 
distribution of exponents over 
multiplication. 

• The student’s explanation displays no 
understanding of the distribution of 
exponents over multiplication. 

• No explanation is provided. 

Explanation Score 


(iv) (n + 3) 2 = n 2 + 3 2 = n 2 + 9 


Accuracy/Precision 

• (1) The student corrects the error by 
adding the term 6n to the RHS. 

• (0) The student does not correct the 
error. 

Accuracy/Precision Score I 


Explanation 

• (3) The student’s explanation displays 
understanding of the expansion of 
squared binomials (e.g. “When you 
expand (n+3) 2 you get (n+3)(n+3)..”). 

• (2) The student’s explanation displays 
limited understanding of the expansion of 
squared binomials. 

• The student’s explanation displays no 
understanding of the expansion of 
squared binomials. 

• No explanation is provided. 


Explanation Score 
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FAL 16, Interpreting Algebraic Expressions 
Common Issues (Misconceptions) Rubric 


Item: Interpreting Expressions-0 1 

Prompt: Write algebraic expressions for each of the following... 


Code 

Common Issue 

Example of Present 
Common Issue 

Is the Common 
Issue present? 
(Yes/No) 

Order of 

Operations/ 

Parentheses 

Student writes expressions 
left to right showing little 
understanding of the order of 
operations implied by the 
symbolic representation. 

or 

Student does not construct 
parentheses correctly or 
expands them incorrectly. 

Qlb. Student writes 4 + nx 

5 instead of 5(n + 4) 

Qlc. Student writes 4 + n + 

5 instead of 



Item: Interpreting Expressions-02a 

Prompt: Imagine you are a teacher. Decide whether the following work is correct or incorrect. If 
you see an error: (a) Cross it out and replace it with a correct answer. 


Code 

Common Issue 

Example of Present 
Common Issue 

Is the Common 

Issue present? 
(Yes/No) 

Order of 

Operations/ 

Parentheses 

Student does not construct 
parentheses correctly or 
expands them incorrectly. 

Any of the following are 
not indicated as incorrect: 

• 2(n + 3) = 2n + 3 

• (5nf = 5n 

• (n + 3) 2 = n 2 + 3 2 
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FAL 22, Finding Equations of Parallel and Perpendicular Lines 

Scoring Exemplar 


FndngjEquatioos of Parallel & PerpentScular Lints Student Materials Beta Version 

Parallel and Perpendicular Lines 


Here are some equations of straight lines 


y*2x = 8 

y=--2*-»-8<l 

2y ♦-Jjr-tl - 0 

y=-i*— pi 

4 y-x -1 

y= Jx ♦ A g 

1 

H 

1 

y-2(x-l) 

y=-2x--2g 

2y - x— 4 

y=-fx-2<» 

y -t 2x •» 2 ■ 0 

y*-2x-24 

y-iur + 2 

y -4 - x 

y-.-X-.-4<| 

2y-4-x 

ys-px-»-24 


1 . Which four lines form the four sides of a rectangle? 

Explain your reasoning carefully. 

A re ctong le has- four- sides-with parolle I- opposite sides. -The slope s-ef- para lie Mine sore equal' and- the- 
prod uct-of • the slope of a- Ime and- itsperpendicu lor- The re ore -two-pa irsof- lines- wrth equal sbpes- 
(thus- theyare -porallel).<| 

•-* y = -2x-*-8'andy=--2x---2'(bothhavesbpejof--2)<l 
y-=-|x-»-2'ond-y=-yx---2 , (both-have sbpes-of t)4 

Be cause I knowthat the prod uct-of - the slope -of -a- Ime and its perpendicubr- is*-l.-and--2-*-p=--l.-I-kn#w 
that-the'2-setsofporalbHinesaboveareperpendicubr- toeoch-o the r.-and- the -4- Ime jformarectongle.<I 

<1 

4 
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FAL 22, Finding Equations of Parallel and Perpendicular Lines 

Quality Rubric 

Item: Parallel and Perpendicular Lines-01 

Prompt: Which four lines form the four sides of a rectangle? Explain your reasoning carefully. 


Accuracy/Precision 

Score one point for each of the 
following correct lines: (0-4 
points possible): 

• y+2x=8 or y=-2x+8 

• y+2x+2=0 or y=-2x-2 

• 2y=x-4 or y= Zix-2 

• y=V2X+2 


Accuracy/Precision Score 


Explanation 

• (3) The student provides a clear, well-supported 
explanation with explicit reference to the 
geometric properties of rectangles and the 
relationships between slopes of parallel and 
perpendicular lines. 

• (2) The student displays some understanding of 
the relevant mathematical concepts (the 
geometric properties of rectangles and the 
relationships between slopes of parallel and 
perpendicular lines), but does not provide a clear, 
well-supported explanation. 

• (1) An explanation is provided, but it does not 
display understanding of the relevant 
mathematical concepts (the geometric properties 
of rectangles and the relationship between slopes 
for parallel and perpendicular lines). 

• (0) There is no explanation. 

Explanation Score 


Item: Parallel and Perpendicular Lines-02 

Prompt: Complete the drawing below to show the four lines and the x- and v-axes. Label the 
lines clearly. 


Accuracy/Precision 

Score one point for each of the following (0-7 points possible): 

• The student correctly draws the missing line, y z=1 /2X+2. 

• The student correctly draws the x-axis. 

• The student correctly draws the y-axis. 

• The student correctly labels y=-2x+8 (y+2x=8 also acceptable) 

• The student correctly labels y=-2x-2 (y+2x+2=0 also acceptable) 

• The student correctly labels y =1 /2X-2 ( 2y=x-4 also acceptable) 

• The student correctly labels y=V2X+2 

Accuracy/Precision Score [ 
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FAL 22, Finding Equations of Parallel and Perpendicular Lines 
Common Issues (Misconceptions) Rubric 


Item: Parallel and Perpendicular Lines-01 

Prompt: Which four lines form the four sides of a rectangle? Explain your reasoning carefully. 


Code 

Common Issue 

Example of Present Common 

Issue 

Is the Common 
Issue present? 
(Yes/No) 

Rectangle 

Properties 

Student does not use 
geometric properties 
of a rectangle in 
solution. 

Student does not mention that a 
rectangle has two pairs of parallel 
sides 

or 

Student does not mention that 
adjacent sides of a rectangle are 
perpendicular. 


Slope 

Properties 

Student does not 
identify parallel / 
perpendicular lines by 
their slopes. 

Student does not select pairs of 
parallel lines 

or 

Student does not select parallel pairs 
that are perpendicular to each other. 


Slope from 
Equation 

Student demonstrates 
limited understanding 
of the link between the 
slope and the form of 
the equation of a 
straight line. 

Student identifies slopes for 
equations in which y is given 
explicitly in terms of x ( y=mx+b ) 
but not for other equations 

or 

Student reads the number in front of 
x as if it were the slope in all 
equations. 



Item: Parallel and Perpendicular Lines-02 

Prompt: Complete the drawing below to show the four lines and the x- andy-axes. Label the 
lines clearly. 


[unscored] 
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Exhibit A3: 

MDC Teacher Survey 2013 
[ login ] 


Before you begin, note that the math tools provided to you by the Gates Foundation are referred to by many names. 

We use the phrase Formative Assessment Lessons (MDC Lessons) to refer to the math tools 
that were developed by the Shell Centre (whose name appears on the course materials) as part of the Mathematic Design Collaborative (MDC). 

You may recall that these tools have four distinct parts: 
the initial or pre-assessment, the collaborative activity; the whole class plenary and the post-assessment. 

Some of the questions in the survey make reference to the Common Core State Standards (CCSS). In different states, this could be referred to 
differently, for example, in the state of Colorado, it is referred to as Colorado Academic Standards. 


You are about to enter the survey. To go back a page, please use the 
survey's red "Back" button, not your browser's back button. 

Your answers will be saved each time you click "Next." 

The survey takes about 30 minutes to complete. You may leave and return multiple times. 

If you do return, after entering your login code, you will be placed in the screen you last visited. 
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Please select the best answer for each question. Some instructions are in italics. 

TEACHER BACKGROUND INFORMATION 


1. What is/are your current position(s)? Please CHECK ALL that apply. 
□ a Classroom math teacher 
□ b Special Education 
□ c Department head 
□ d Math coach 

□ e Other (please specify) [ 100 characters ] 


2 . At which grade level(s) do you teach? Please CHECK ALL that apply. 
□a Middle school (6 th - 8 th grade) 

□ b High school (9 th - 12 th grade) 


3. In what content area(s) do you teach? Please CHECK ALL that apply. 
□ a Pre-Algebra 
□ b Algebra I 
□ c Geometry 
□ d Algebra II 
□ e Pre-Calculus 
□ f Calculus 

□ g Probability and Statistics 

□ h Other (please specify) [ 100 characters ] 
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4. To the nearest year, how long have you ... 



a) ... been a teacher? 

year(s) [ 

integer, 0-99 ] 


b) ... taught in your current school? 

year(s) [ 

integer, 0-99 ] 


c) ... taught in your current district? 

year(s) [ 

integer, 0-99 ] 



Yes 

No 

5. 

Do you teach ELL students? 

o 2 

Oo 

6. 

Do you teach special education students? 

O 

O 

7. 

Do you teach students who are struggling in Math? 

O 

O 

8. 

Do you teach students with advanced mathematical levels? 

O 

O 


9. How would you describe your participation in the MDC initiative? 

01 Required 

0 2 Voluntary 

0 3 I have not taught an MDC Lesson in 2012-2013 [ End of survey; go to Reg Close ] 

0 4 I opted out of participating in 2012-2013 (please specify a reason for opting out) [ 1000 characters ] [ End of 

survey; go to Reg Close ] 


10. Is this your first year in the MDC initiative? Oi Yes [skiptoQ12] O 0 No 


11. How many MDC Lessons did you teach last year (2011-12)? [ integer, 0-99 ] MDC Lesson(s) 

Please enter a 0 if you did not teach any MDC Lessons last year (2011-12). 

12. How many MDC Lessons will you have taught in total during the current school year (2012-13)? [ integer, 0-99 ] MDC Lesson(s) 
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13 . My involvement with the MDC initiative has included the following activities: Please CHECK ALL that apply. 
□ a Teaching the MDC Lessons 

□ b Aligning MDC Lessons to the course content/pacing guide 

□ c Coaching others on how to use MDC Lessons 

□d Presenting at an MDC Lesson professional development session 


14 . How many MDC Lessons do you have access to? Check ONLY ONE that applies 

01 1-5 MDC Lessons 

0 2 6-10 MDC Lessons 

0 3 11-20 MDC Lessons 

0 4 more than 20 MDC Lessons 
O s Don't know 
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15. How do you access the MDC Lessons? Please CHECK ALL that apply. 


01 They are emailed to me by district administrator. 

0 2 They are saved on the district website. 

0 3 They are on the Shell Centre Website. 

0 4 Hard copies are provided during professional development. 

0 5 Other 


SUPPORT FOR USING MDC LESSONS 


16. Indicate whether the following people visited your classroom when you were teaching an MDC Lesson. 
[ randomize options, letters don't appear ] 




Visited 

Did not visit 

a. 

District or network MDC project lead 

o. 

Oo 

b. 

Principal 

O 

O 

c. 

Instructional coach/department head 

O 

O 

d. 

Teacher colleague 

O 

O 
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BELIEFS ABOUT TEACHING MATHEMATICS SKILLS 

Q17 is about teaching mathematics. 

17. Please indicate the degree to which you agree or disagree with the following statements: 



Agree 

Agree 

Somewhat 

Disagree 

Somewhat 

Disagree 

1 believe the following strategies are effective ways of strengthening my students' 
mathematical understanding: 





a. Teacher taking on the role of "facilitator" or "coach" 

o. 

o 2 

o 3 

o 4 

b. Peer-to-peer problem-solving 

o 

o 

o 

o 

c. Asking students guiding questions 

o 

o 

o 

o 

d. Providing class time for students to persevere through difficult math problems 

o 

o 

o 

o 

PURPOSE OF INITIATIVE 

18. Please indicate the degree to which you agree or disagree with the following statements: 





Math MDC Lessons are effective in... 





a. ... improving students' ability to think mathematically. 

o 

o 

o 

o 

b. ... providing a curricular resource for teachers in addressing the Common Core State 





Standards. 

o 

o 

o 

o 

c. ... encouraging teachers to adjust their pedagogy in math instruction from a focus on 





process to a focus on building conceptual understanding. 

o 

o 

o 

o 

d. ... making instruction more engaging for students. 

o 

o 

o 

o 

e. ... using formative assessment to identify student strengths and weaknesses and to 





inform instruction. 

o 

o 

o 

o 
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TEACHER PERCEPTIONS OF TOOL UTILITY 


Yes No 

19. Please indicate whether using the MDC Lessons has helped you in the following ways during MDC Lesson use. 

During MDC Lesson instruction, using the MDC Lessons has helped me ... 


a) 

... find effective strategies for teaching my subject content. 

Oi 

o 

b) 

... learn new ways to include formative assessment in my classes. 

o 

o 

c) 

... learn detailed information about my students' mathematical strengths and weaknesses. 

o 

o 

d) 

... provide students with more detailed feedback about their work. 

o 

o 

e) 

... implement the Common Core State Standards. 

o 

o 

f) 

...create an environment that promotes mathematical discourse. 

o 

o 

g) 

... better engage students. 

o 

o 
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Agree Disagree 

Agree Somewhat Somewhat Disagree 

20. Please indicate the degree to which you agree or disagree with the following statements. 

MDC Lessons help me differentiate instruction ... 

[ skip next item if Q5 = no ] 

a. ... for ELL students. Oi 0 2 0 3 0 4 

[ skip next item if Q6 = no ] 

b. ... for special education students. O O O O 

[ skip next item if Q7 = no ] 

c. ... for students who struggle with math. O O O O 

[ skip next item if Q8 = no ] 


d.... for students with advanced math abilities. 


o o o o 


21. The MDC Lessons are flexible enough to fit the needs of all my students. 


o o o o 
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Question 22a-c is about the most recent MDC Lesson you taught. 


22. Please indicate the degree to which you agree or disagree with the following statements: 


a. Based on the information collected from using the MDC Lesson, I adjusted my 

instruction to meet the needs of individual students. 

b. I knew where to fit the MDC Lesson in the unit. 

c. The pre-assessment revealed my students' misconceptions to me. 


23. Please indicate the degree to which you agree or disagree with the following statements: 

a. Using the MDC Lessons raised my expectations for students' mathematical work. 

b. The MDC Lessons have become an important part of my instructional practice. 


Agree Disagree 

Agree Somewhat Somewhat Disagree 


Oi 

o 2 

o 3 

o 4 

o 

o 

o 

o 

o 

o 

o 

o 


o o o 

o o o 


24. Select the phrase that best completes the sentence: 

I use the MDC instructional strategies ... 

Oi ...often 0 2 ...sometimes 0 3 ...rarely 

... during non-MDC instruction. 


STUDENT IMPACT 

25. Compared to my usual instruction, during the MDC Lessons, my students ... 

Oi ... are more engaged. 0 2 ... show the same level of engagement. 0 3 ... are less engaged. 
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26. Please indicate the degree to which you agree or disagree with the following statements. 


Agree Disagree 

Agree Somewhat Somewhat Disagree 


The MDC Lessons have improved my students' mathematical reasoning. 

o 1 

o 2 

o 3 

o 4 

The MDC Lessons are supporting my students' college-readiness. 

o 

o 

o 

o 


Question 27a-c asks you to reflect back on your most recent experience implementing an MDC Lesson during the current school year (2012-13). 

27a. When I taught the most recent MDC Lesson, the majority of my students improved their content knowledge. 

Qi Yes O 0 No 

27b. When I taught my most recent MDC Lesson, the majority of my students improved their conceptual understanding. 

Oi Yes Q 0 No 


TOOL USE (FOI Questions) 


28. To what extent did you follow the MDC Lessons Teacher Guide when implementing instruction? 

01 Completely 

0 2 Mostly 

0 3 Somewhat 

0 4 Not at all 

29. How many class periods did you spend implementing a typical MDC Lesson in the current school year (2012-13)? class periods 

[ integer, 1-99 ] 
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30 . What percent of class time did you spend on each of the following MDC components during a typical MDC Lesson in the current school 
year ( 2012 - 13 )? 

Pre-assessment [Enter number] 

Small-group collaborative work [Enter number] 

Whole-class plenary/discussion [Enter number] 

Post-assessment [Enter number] [limit the sum of the numbers in this ques to 100] 

31 . About how long did you spend reviewing your class's answers to the MDC pre-assessment? minutes [ integer, 1-99 ] 

32 . Please indicate, across all the MDC lessons you taught in the current school year (2012-13), how frequently you responded to the pre- 
assessment in the following ways. 


a. Little feedback provided because of time constraints 

b. Wrote comments on most individuals' pre-assessments 

c. Wrote questions on most individuals' pre-assessments 

d. Reviewed common errors at start of class 


Almost always 

Often 

Sometimes 

Almost never 

o. 

o 2 

o 3 

o 4 

o 

o 

o 

o 

o 

o 

o 

o 


33. Please select the phrase that best reflects your practice: 

When students are working on the collaborative portion of the FALs, ... 

01 ... I typically let students choose their own partners or group. 

0 2 ... I typically assign students to homogeneous math ability groups. 

0 3 ... I typically assign students to heterogeneous math ability groups. 
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34. Please indicate the degree to which you use the following strategies while implementing the collaborative activity portion of your MDC 
Lessons in the current school year (2012-13). 

[ randomize options, letters don't appear ] 




Always 

Often 

Sometimes 

Never 

a. 

observed and listened but did not offer feedback 

o 3 

o 2 

o 3 

o 4 

b. 

offered hints and suggestions 

o 

o 

o 

o 

c. 

raised questions 

o 

o 

o 

o 

d. 

provided the answer 

o 

o 

o 

o 

e. 

asked another student to explain the concept 

o 

o 

o 

o 

f. 

spent time re-teaching the concept 

o 

o 

o 

o 


Q35-38 are about the most recent MDC Lessons you taught. 

35. Reflecting on the most recent MDC Lesson, when did students complete the post-assessment? 

01 In class, at the end of the plenary/class discussion 

0 2 As homework 

0 3 Other (please specify) [ 1000 characters ] 

0 4 I did not use the post-assessment 

36. Which one of the following reasons best describes why you did not use the post-assessment in your most recent MDC Lesson? [Only 
displayed if teacher selects 4 on Q35] 

01 I ran out of time. 

0 2 It seemed redundant. 

0 3 My students understood the material. 

0 4 It was too hard. 

0 5 Other (please specify) [ 1000 characters ] 
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37. What percent of your students do you think gained a strong grasp of the material by the end of the MDC Lesson? 

01 100% 

0 2 75-99% 

0 3 50-74% 

0 4 25-49% 

0 5 5-24% 

0 6 <5% 

0. 99 Not sure 


38. How effective were each of the following components of the MDC Lesson in promoting student learning? 

Very 

Effective Effective 


Somewhat Not 
Effective Effective 


a. 

Pre-assessment 

o. 

o 2 

o 3 

o. 

b. 

Small-group collaborative work 

o 

o 

o 

o 

c. 

Whole Class plenary/discussion 

o 

o 

o 

o 

d. 

Post-assessment 

o 

o 

o 

o 
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POTENTIAL BARRIERS TO TOOL USE 

39. Please indicate the degree to which you agree or disagree with the following statements. 





Agree 

Disagree 



Agree 

Somewhat 

Somewhat 

a. 

1 had sufficient time to plan for the lessons. 

o. 

o 2 

0 3 

b. 

1 felt adequately prepared to effectively use the lessons. 

O 

O 

O 

c. 

1 had sufficient time to prepare for the pre-assessment. 

O 

O 

O 

d. 

Using the MDC Lessons takes too much time away from covering required 
curriculum topics. 

O 

O 

O 

e. 

The preparation required for the MDC Lesson collaborative activity is an 





impediment to using the lesson. 

O 

O 

O 

f. 

During the collaborative portion of the MDC Lesson, it is difficult for me to 





interact with every group. 

O 

O 

O 


Disagree 

0 4 

O 

o 

o 

o 

o 


85 




SCALING OF MDC INITIATIVE 


Q40 is about using the MDC Lessons next year ( 2013 - 14 ). 


40 . Please indicate the degree to which you agree or disagree with the following statements: 



Agree 

Agree 

Somewhat 

Disagree 

Somewhat Disagree 


a) 

1 would like to have access to more MDC Lessons next year. O a 

o 2 


o 3 


o 4 


b) 

1 look forward to teaching MDC Lessons next year. O 

o 


o 


o 


c) 

1 plan to improve how 1 teach the MDC Lessons next year. O 

o 


o 


o 


41 . 

Please indicate the degree to which you agree or disagree with the following statements below: 






a) 

My participation in the MDC initiative is worth the time and effort involved. 

o 

O 


o 


o 

b) 

1 see the ideas and practices of the MDC initiative gaining traction in my school. 

o 

o 


o 


o 

c) 

1 have noticed an increase in the number of teachers using the MDC Lessons in my 








school since last year. 

o 

o 


o 


o 

d) 

There are other curricular initiatives or programs in the district that address some of 








the same purposes as MDC. 

o 

o 


o 


o 

e) 

The other curricular initiatives or programs in the district create competing priorities 








with the MDC initiative. 

o 

o 


o 


o 

f) 

The district has the commitment to sustain the MDC initiative. 

o 

o 


o 


o 
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42. Have you shared any of your MDC Lessons with a teacher who is not participating in the MDC initiative? Oi Yes 


Q 0 No 


SCHOOL LEADERSHIP 


Q43 is about administrators at your school and district. 


43. Please indicate the degree to which you agree or disagree with the following statements. 



My school administrators ... [ randomize options, letters don't appear ] 

Agree 

Agree 

Somewhat 

Disagree 

Somewhat 

Disagree 

Don't 

Know 

a. 

... have a firm understanding of the MDC Lessons. 

o. 

o 2 

o 3 

o 4 

O.99 

b. 

... have made formative assessment a priority at my school. 

o 

o 

o 

o 

o 

c. 

... encouraged me to participate in the MDC initiative. 

o 

o 

o 

o 

o 

d. 

... have provided me with feedback about my instruction of the MDC Lessons. 

o 

o 

o 

o 

o 

e. 

... have provided ongoing support for the implementation of the MDC Lessons. 

o 

o 

o 

o 

o 

f. 

... expressed concerns that the MDC Lessons are taking time away from other 
instructional priorities. 

o 

o 

o 

o 

o 

g- 

... have attended professional development about the MDC Lessons. 

o 

o 

o 

o 

o 

h. 

... have communicated how the MDC Lessons are aligned with other 







school initiatives. 

o 

o 

o 

o 

o 

My district administrators ... [ randomize options, letters don't appear ] 






i. 

... support the MDC initiative. 

o 

o 

o 

o 

o 

j- 

... encourage my participation in the MDC initiative. 

o 

o 

o 

o 

o 

k. 

... provide ongoing support for the implementation of the MDC Lessons. 

o 

o 

o 

o 

o 

1. 

... have a firm understanding of the MDC Lessons. 

o 

o 

o 

o 

o 

m. 

. ...have attended professional development about the MDC Lessons. 

o 

o 

o 

o 

o 
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ALIGNMENT 


44. 

Please indicate the degree to which you agree or disagree with the following statements. 

Agree 

Agree 

Somewhat 

Disagree 

Somewhat 

Disagree 

a. 

The MDC Lessons align well with my school's curriculum. 

o 1 

o 2 

0 3 

o 4 

b. 

The MDC Lessons help prepare my students for the current state assessment(s). 

O 

O 

O 

O 

c. 

The MDC Lessons align with the Common Core State Standards. 

O 

O 

O 

O 

d. 

1 see the unique value of the MDC Lessons to address the Common Core State Standards. 

O 

O 

O 

O 

COLLABORATION 






Questions 45-46 are about your interactions with your MDC colleagues. 


45. Do you and your MDC colleagues have regularly scheduled common planning time to discuss MDC? 
Oi Yes O 0 No 


46. I would describe my MDC colleagues as collaborative. 


Agree Disagree 

Agree Somewhat Somewhat Disagree 

Oj O2 O3 O4 
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47. About how often do you have scheduled meetings (as opposed to informal discussions) with your MDC colleagues to discuss student 
work, instructional strategies, or teaching approaches? [skip if Q45=no] 

01 At least once a week 

0 2 Every other week 

0 3 Once a month 

0 4 Once per quarter/trimester/semester 

0 5 Never 

48. About how often do you have informal discussions (as opposed to scheduled meetings) with your MDC colleagues to discuss 
student work, instructional strategies, or teaching approaches? 

0 : At least once a week 

0 2 Every other week 

0 3 Once a month 

0 4 Once per quarter/trimester/semester 

0 5 Never 


Agree 

49. Collaboration with my MDC colleagues helps me to ... [ randomize options, letters don't appear ] 

Agree 

Somewhat 

Disagree 

Somewhat 

Disagree 

a. ...more effectively use the MDC formative assessment strategies. 

o. 

o 2 

0 3 

o 4 

b. ...better support student learning. 

O 

O 

O 

O 

c. ...use the MDC Lessons in my class. 

O 

O 

O 

O 

d. ...review pre-assessments. 

O 

O 

O 

O 

e. ...facilitate collaborative group work. 

O 

O 

O 

O 

f. ...facilitate the plenary or whole-class discussion. 

O 

O 

O 

O 

g. ...identify common math misconceptions. 

O 

O 

O 

O 

h. ...determine where to use an MDC Lesson in a unit. 

O 

O 

O 

O 

i. ...develop feedback questions. 

O 

O 

O 

O 

j. ...determine how to group students. 

O 

O 

O 

O 

k. ...review post-assessments. 

O 

O 

O 

O 
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WORKING WITH EXPERIENCED MDC COLLEAGUES 


50. Are there teachers in your school or district who used the MDC Lessons last year (2011-2012)? Oi 

51. Did you work with a colleague more experienced with MDC this year (2012-2013)? Oi 


Yes 

O 0 

No 

[ skip to 

Q53 ; 

Yes 

O 0 

No 

[ skip to 

Q53] 


52. How much did working with an experienced MDC colleague help you to use MDC Lessons? 

01 a great deal 

0 2 a fair amount 

0 3 some 

0 4 not much 
Q 5 not at all 


PROFESSIONAL DEVELOPMENT 


53. Have you participated in formal professional development sessions related to MDC during the current school year (2012-13)? 


Oi Yes 


O 0 No [ skip to Q61] 
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54. Which PD providers facilitated the MDC professional development you attended this year (2012-13)? Please CHECK ANY that applies 

□ a State or regional staff 

□ b External partner (e.g., SREB, Ann Shannon, Math Solutions) 

□ c District or network staff 
□ d School-based staff 

□ e I don't remember who facilitated the PD this year 

55. How many formal, scheduled MDC professional development sessions have you attended this year (2012-13)? [integer, 1-99 ]_ 

session(s) 

56. Please indicate whether you participated in the following types of MDC professional development sessions. 



Participated 

Did not participate 

a. One-on-one classroom visits 

Ob 

O 0 

b. Coaching 

o 

o 

c. Webinars 

o 

o 

d. Small group meetings 

o 

o 

e. School-wide meetings 

o 

o 

f. District-wide meetings 

o 

o 

g. Cross-district meetings 

o 

o 
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57. Please indicate whether the MDC professional development sessions you participated in was effective or not effective. 

Not 

Effective Effective 

a. [ fill choice from previous item here ] Qi O 0 

b. [ fill choice from previous item here, etc., etc. ] O O 


58. Please indicate whether the MDC professional development sessions you have participated in contained the following types of content: 


a. Using MDC Lessons as a way to implement the Common Core State Standards 

b. Administering the pre-assessment 

c. Identifying common misconceptions through the pre-assessment results 

d. Developing feedback questions 

e. Facilitating small group work with guiding questions 

f. Facilitating whole group plenary discussions 

g. Placing lessons at the appropriate point in the unit 

h. Determining how to use the post-assessment results to guide continued instruc 

i. Differentiating MDC lesson instruction to meet student needs 

j. Implementing MDC lessons with ELL students. 

k. Implementing MDC lessons with special education students. 

l. Implementing MDC lessons with students who are struggling with Math. 

m. Implementing MDC lessons with students with advanced mathematical levels. 


PD contained 

PC did not contain 

this content 

this content 

o 1 

O 0 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

unit O 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 


92 



Q59 asks about MDC professional development that would support your implementation of the MDC initiative. 


Yes 

59. Please indicate whether you would like more professional development on ... 

a. ...using MDC Lessons as a way to implement the Common Core State Standards Ox 

b. ...administering the pre-assessment. O 

c. ...identifying common misconceptions through the pre-assessment results. O 

d. ...developing feedback questions. O 

e. ...facilitating small group work with guiding questions. O 

f. ...facilitating whole group plenary discussions. O 

g. ...placing lessons at the appropriate point in the unit. O 

h. ... determining how to use the post-assessment results to guide continued instruction in the unit. O 

i. ...differentiating lesson instruction to meet student needs O 

j. ...implementing lessons with special education students O 

k. ...implementing lessons with ELL students O 

l. ...implementing lessons with students who are struggling in Math O 

m. ...differentiating lesson with students with advanced mathematical levels O 


No 


O 

O 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 


60. Are you compensated for attending professional development sessions? Oi Yes 


O 0 No 


61. What additional supports and training would help you use the MDC Lessons? Please use the field below to describe. 


[ limit 1000 characters ] 
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62. Surveys are not perfect. Maybe we missed some things that you think are important about the Gates MDC initiative. Below, we invite 
you to write your assessment and comments about the initiative as you have experienced it. 

[ limit 1000 characters ] 


These last few questions are about you. 

63. Are you certified to teach mathematics? Oi Yes O 0 No 

64. Are you a member of National Council of Teachers of Mathematics (NCTM)? Oi Yes O 0 No 

65. What is your race/ethnicity? Please CHECK ONE that apply. 

□ a Native American 
□ b Asian/Pacific Islander 
□ c Black or African American 
□d Hispanic or Latino 
□ e White or Caucasian 
□ f Multiracial 

□ g Other (please specify) [ 100 characters ] 
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[ Go to "Regular Close" ] 


REGULAR CLOSE 

Thank you very much for the time and thought you have put into completing this survey. 

To ensure anonymity, your responses will be combined with those from teachers of numerous schools. 

Your responses will help to inform implementation of the Math Design Collaborative. 

DON'T AGREE CLOSE 

We are sorry you have chosen not to participate in the survey. 

Thank you for visiting Research for Action's and the National Center for Research on Evaluation, Standards and Student Testing's survey on the 

Gates Math initiative. 

ERROR MESSAGE IF AN ANSWER IS LEFT BLANK: 

You have not given an answer for a question on this screen. 

Do you want to go back to give an answer or continue with the survey? 

O I want to go back to answer the question. 

O I want to continue without answering the question. 
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Exhibit A4: MDC CRESST Assessment: Short Teacher Survey 


1. What date(s) were the enclosed assessments administered? 

2. How much total time did students spending taking the assessments? 

minutes 

Please answer the following questions about your instruction in the 2012-13 school year. 
Answers should reflect your instruction for the classroom in which you are administering 
the enclosed assessments. 

3. Place an X in the box next to each Formative Assessment Lesson (FAL) you 
taught this school year in this class. (For your convenience, the lesson number is 
included in parentheses where applicable.) 



Manipulating Radicals (40) 


Mean, Median, Mode, and Range 


Modeling Situations with Linear Equations 
(5) 


Modeling: Having Kittens (39) 


Modeling: Rolling Cups (17) 


Operations and Radicals 


Optimization Problems: Boomerangs (1) 


Rational and Irrational Numbers 1 (27) 


Rational and Irrational Numbers 2 (3 1) 


Representing Data Using Box Plots (26) 


Representing Data Using Frequency 

Graphs (25) 


Representing Polynomials (41) 


Solving Linear Equations in One Variable 
(70) 


Solving Linear Equations in Two Variables 

(3) 


Solving Quadratic Equations: Cutting 
Comers (43) 


Sorting Equations and Identities (13) 



Comparing Investments (37) 


Creating and Solving Equations (44) 


Defining Regions Using Inequalities (14) 


Equations of Circles 1 (34) 


Estimating and Approximating: The 

Money Munchers (15) 


Evaluating Statements about Probability 
(21) 


Ferris Wheel (23) 


Finding Equations of Parallel and 
Perpendicular Lines (22) 


Forming Quadratics (20) 


Functions and Everyday Situations (46) 


Generalizing Patterns: Table Tiles (9) 


Interpreting Algebraic Expressions (16) 


Interpreting Distance-Time Graphs (2) 


Interpreting Statistics: A Case of 

Muddying the Waters (12) 


Lines and Linear Equations (62) 


Manipulating Polynomials (45) 
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4. How many FALs which do not appear on the list below did you teach in this 

class? 

5. Please indicate the degree of emphasis you placed on each of the following 
standards during your 2012-13 instruction in this class, by placing an X in the 
appropriate column. 



No 

Emphasis 

Slight 

Emphasis 

Moderate 

Emphasis 

Sustained 

Emphasis 

a. Interpret the structure of expressions. 





b. Rewrite rational expressions. 





c. Write expressions in equivalent forms 
to solve problems. 





d. Create equations that describe numbers 
or relationships 





e. Solve systems of equations 





f. Use coordinates to prove simple 
geometric theorems algebraically 





g. Analyze functions using different 
representations 






6. If there were any problems with the administration of the assessments, or you 
have any other comments, please use the space below. 
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Exhibit A5: CRESST Math Assessment 



NATIONAL CENTER FOR RESEARCH ON 
EVALUATION. STANDARDS. & STUDENT TESTING 


MATHEMATICS 

ASSESSMENT 




National Center for Research 

on Evaluation, Standards, & Student Testing 

UCLA I Graduate School of Education A Information Studies 


BlLLe^MELINDA 

GATES foundation 


© 2013 Regents of the University of California 
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ALGEBRA 


Instructions 

Read and answer the following questions. If you are asked to explain something, make sure you 
use words and not only calculations in your explanation. 


1) Which of the following statements is true about this equation? 

2(x - 4) = 2v - 8 

@ This equation has exactly one solution. 

® This equation has exactly two solutions. 

© This equation has no solutions. 

© This equation has an infinite number of solutions. 


2) Which equation does not have a solution? 

© x= = (x + 2Xx-2) 

© (x | 2 = X 2 

© (2x - 3Rx - 2) = 0 
© x= 5 


100 






ALGEBRA 


3) On the line next to each equation, give the solution to the equation or write "no solution” 
if the equation has no solution. 

a) Equation 1 : x-1 = 1 - x 

b) Equation 2: x 2 = (.v + 5)(x - 5) 

4) Write a number in the box below to make this equation an identity. 

5(x - 3) + 3 = 3x - 1 | + 2x 

5) Explain why the expression y 2 + 6 2 is not equivalent to the expression 0’ + 6) 2 . 
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ALGEBRA 


6) Which expression is equivalent to the expression (.v + 3) ! + 2*(.v - 1)? 

@ 3;e + 8 
© 3*2 + 6 * + 8 
© 3*~ - 2* + 9 
© 3*2 + 4* + 9 


7) Which expression is equivalent to the expression 8/i - 24? 

@ 4(2n - 6) + 2 

® (n + 2)2 + 4<« - 7) 

r~\ 5« — 40 1 4 n 

© + — (lln) - — 

© _^L_ 4 ,n) + ^-+l 

^ 2 n 


8) Your friend Jay says lhat this area diagram corresponds to expression .v 3 + 4* + 4. 

x 2 


.v 

2 


Shade the box (or boxes) that correspond to the 4* term. 
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ALGEBRA 


9) Brian makes .v dollars working at a car wash. Brian's mom agreed to match the amount he 
earns at the car wash if Brian mows the lawn. Brian's dad agreed to give Brian three dollars 
if Brian washes dishes after dinner. Write an expression for the total amount of dollars Brian 
earns for working at a car wash, mowing the law n, and washing dishes in terms of x. 


10) Xocllc is an animal lover w ho has d number of dogs and c number of cats. Xoelle has 2 more 
cats than dogs. 

Using that information, determine which of the tsvo expressions below is larger and CIRCLE 
that expression. Explain your reasoning on the lines below the expressions. 

c + d and 2c 



5 
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ALGEBRA 


Use the scenario below to answer questions 11-13. 


A store sells popsicles for S3 and chocolates for $4. 

p = numbers of popsicles sold 
c = number of chocolates sold 

The following equations are true: 

Equation 1: 6 c = p 
Equation 2: 3p + 4c = 88 


1 1 ) On the lines below, explain the meaning of Equation 2. 


12) Which of the follow ing sets of values satisfies Equation 1? 

(a) c= 12, p = 2 
® c = 4, p = 19 
© c = 3, p =18 
© c = 13, p = 12 


13) Which of the follow ing sets of values satisfies Equation 1 and Equation 2? 

© c = 4, p = 20 
(b) c = 3 , p =18 
© c = 5, p =30 
® c = 4, p = 24 


CONTINUE 


104 






ALGEBRA 


14) Write an equation for the following statement: 

There are 6 limes as many rabbits as foxes in the forest, 

Use r for the number of rabbits and/for the number of foxes in your equation. 

15) Determine the values of x andy that simultaneously satisfy both of these equations. 

x + 4 = 3 y 
6x - 4y = 32 

* = y- 

16) Tliey-intercept of liney = -3 a- - 5 is . 


17) 



y=— v + 2 


The lines represented by these equations: 


(a) are parallel 

(5T) are the same line 

© are perpendicular 

@ intersect at an angle other than 90° 



Z1 


7 
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ALGEBRA 


3’ ~ 8 = -j-x 2y - -yx = 3 

The lines represented by these equations: 

© are parallel 
© are the same line 
© are perpendicular 
© intersect at an angle other than 90° 

19) a) The slope of line 5y + 3x= 15 is 

b) Explain how you determined the slope of the line. 
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ALGEBRA 


20) Write the equation for the line that passes through (-1, 5) and is perpendicular to line 
.v + 4y = - 6. 


21) What is the vertex of the graph for the quadratic equation below? 

y = x 1 + 6.x - 7 

© (0,-7) 

© (5,0) 

© (-3,-16) 

© (9,-12) 

22) Ava serves a volleyball. The height of the ball can be described by h = -t* + It + 13, where h 
represents the height of the volleyball (in feet) and i represents time (in seconds). 

What is the maximum height that the volleyball reaches? 

© 10 feet 
® 12 feet 
© 14 feet 
© 1 6 feet 
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ALGEBRA 


23) Tilt roots of equation y = .v 2 + .v 6 are ( 


) and [ 


24) The vertex of equation *’ = "2(.v 5) ! 9is( 


Use these 4 quadratic equations and 4 sketches of the graphs of quadratic equations to answer 
question 25. 

e ?- -**+* + 12 @y = (x 6)(x + 1 ) © v=(x + 3)(x 4) ©y = (* 6) 2 + 1 



Graph 1 Graph 2 Graph 3 Graph 4 


25) Equation A matches Graph because: 


to 
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Exhibit A6: CRESST Math Assessment Explanation Rubrics 


Question 5 

Explain why the expression y 2 + 6 2 is not equivalent to the expression (y + 6) 2 . 


Sample Answer: y 2 + 6 2 simply takes both terms and squares them, whereas (y + 6) 2 squares not only both terms 
but results in an additional term. Instead of just squaring the y and the 6, (y + 6) 2 is equivalent to (y+6)(y+6). 
Therefore, while y 2 + 6 2 simplifies to y 2 + 36, (y + 6) 2 simplifies to y 2 + 12y + 36. 


Score 

Description 

Notes 

2 

Correct and complete explanation 

Example: 

The expressions are not equivalent because the 
parentheses around y + 6 in the expression ( y + 
6) 2 indicate that the entire expression y + 6 is 
squared, and that results in v 2 + 1 2 v + 6 2 or v 2 + 
12 y + 36, not y 2 + 6 2 . 



Notes: 

It is not sufficient for students to just point out 
surface feature differences without explaining how 
those surface feature differences impact the 
expressions. 

1 

Partially correct explanation/incomplete explanation 
or an explanation that isn’t completely clear 

Example: 

(y + 6) 2 means that the whole expression is 
squared. 



Example: 

The exponent is outside the parentheses in the 
expression (y + 6) 2 , while the exponent is after each 
term iny 2 + 6 2 . 

0 

Incorrect, too unclear, or too incomplete 

Example: 

(y + 6) 2 has parentheses while y 2 + 6 2 does not 
have parentheses. 


* If students leave a question blank or write ? or IDK, write “BL ” as the score. If students used 
primarily numbers and/or symbols (instead of words) to correctly show their procedure, write “num 
If students incorrectly show their procedure with numbers, count that as a 0. 
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Question 10 


Noelle is an animal lover who has d number of dogs and c number of cats. Noelle has 2 more 
cats than dogs. 

Using rhar information, determine which of the two expressions below is larger and CIRCLE 
that expression. Explain your reasoning on the lines below the expressions. 

c + d and 2c 


Sample Answer :2c should be circled. If Noelle has 2 more cats than dogs, that means that d = c - 2. Therefore 
c + d is equivalent to c + c - 2, which results in 2c - 2, which is smaller than 2c. 


This question wasn’t written with 2 distinct parts, but we will score them that way. Part a will be the score that 
indicates whether they circled the correct expression. Part b will be the score for their explanation. 

a) Only 2c should be circled. Give 1 point if it is circled. 0 for all other answers. 
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b) 


Score 

Description 

Notes 

2 

Student correctly 
explains how he/she uses 
the information about 
Noelle having 2 more 
cats than dogs to 
substitute values in the 
expressions to correctly 
determine that c + d is 
larger (this may or may 
not involve using 
equations) 

OR 

Student uses another 
method to clearly explain 
how he/she determined 
that c + d is larger 

Note: 

A student should be clear about how he/she has used the information 
about Noelle having 2 more cats than dogs. If the student explains a 
method of substitution, the student may explain how this information 
was used in two different ways. . .by substituting into c or d in the two 
expressions. 

Example: 

We know that d = c - 2. Therefore c + d is equivalent to c + c - 2, which 
results in 2 c - 2. That is smaller than 2c, so 2c is the larger expression. 

Example: 

We know that c = d + 2. Therefore c + d is equivalent to d + 2+ d, which 
results in 2d + 2. If we substitute d + 2 for c in 2c, we get 2 (d + 2) or 2d 
+ 4. Because 2d + 4 is larger than 2d + 2, 2c is larger than c + d. 

Example: c 

c + d is the same as c + c - 2 or 2c - 2, which is smaller than 2c. So 2c is 
larger than c + d. 

Example: 

If Noelle has 2 more cats than dogs, that means that c (the number of 
cats) is always larger than d (the number of dogs), so 2c, which is c + c, 
will always be larger than c + d. (Note: it’s sufficient if the student 
doesn’t break 2c down into c + c) 

1 

Student gives a partially 
correct explanation, an 
explanation that isn’t 
completely clear, or an 
incomplete explanation 

Example: 

c + d is equivalent to c + c - 2, which results in 2c - 2. 

Note: 

If the student is on the right path with his/her explanation but makes a 
minor mistake, he/she can also receive a 1. 

Note: 

If the student explains how he/she plugs in numbers to find out which is 
bigger, he/she could get partial credit if they clearly explain the 
relationships between the expressions 

0 

Incorrect, too unclear to 
understand, or too 
incomplete 

Note: 

If the student just reiterates information from the question (e.g., 

“because Noelle has 2 more cats than dogs”), that’s a 0. 


* If students leave a question blank or write ? or IDK, write "BL ” as the score. If students used primarily 
numbers and/or symbols (instead of words) to correctly show their procedure, write “num If students 
incorrectly show their procedure with numbers, count that as a 0. 
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Question 11 


A store sells popsicles for $3 and chocolates for $4. 

p = numbers of popsicles sold 
c = number of chocolates sold 

The following equations are true: 

Equation 1: 6c=p 
Equation 2: 3p + 4c = 88 


On the lines below, explain the meaning of Equation 2. 

Sample Answer: Equation 2 states that the total amount of money earned in sales of p number of popsicles at $3 
and c number of chocolates at $4 was $88. 


Score 

Description 

Notes 

2 

Student must be clear: 

About the relationship between 3 and p 
and the relationship between 4 and c 

AND 

About operations (addition and equals 
signs) between terms 

Note: 

Make sure students are clear on the multiplicative relationships 
(e.g ,,p number of popsicles at $3 or 3 times the number of 
popsicles, not 3 popsicles). Also, they don’t have to explicitly 
mention the addition and equals signs, but the relationships should 
be clear in their explanation. 

Note: 

Although it is not critical that the student specifies that 88 is 88 
dollars, the student’s explanation needs to be clear that 88 is the 
total or the result of summing the other terms. 

Example: 

Equation 2 states that the total amount of money earned in sales of 
p number of popsicles at $3 and c number of chocolates at $4 is 
$88. 

1 

Partially correct explanation/incomplete 
explanation or an explanation that isn’t 
completely clear 

Example: 

The total amount of money earned from 3 popsicles and 4 number 
chocolates is $88. 

Example: 

3 times p plus 4 times c equals 88 

Example: 

Y ou add the amount earned from p number of popsicles and c 
number of chocolates. 

0 

Incorrect, too unclear, or too 
incomplete 

Example: 

3 popsicles and 4 chocolates 


* If students leave a question blank or write ? or IDK, write "BL ” as the score. 
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Question 19 


a) The slope of line 5y + 3.v = 15 is 

b) Explain how you determined the slope of the line. 


Rubric for part a): 


Score 

Description 

Notes 

1 

Students must answer -3/5 


0 

Incorrect or too unclear to 
understand 

Note: (-3/5)x is incorrect. 


* If students leave a question blank or write ? or IDK, write “ BL ” as the score. 
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Rubric for part b): 


Note: Students should have answered -3/5 for part a, so they may not specifically mention -3/5 in their answer to 
part b. That’s fine as long as it’s clear what they were talking about. 

Sample Answer: The equation can be expressed in slope-intercept form, y = mx + b, with m representing the 
slope. In that form, the line is y = (-3/5)x + 3. Therefore, -3/5 is the slope. 


Score 

Description 

Notes 

2 

Student 
gives a 
correct and 
complete 
explanation 

Note: 

Students don’t necessarily have to use the term “slope-intercept form”, if they are clear 
about what they mean. 

Note: 

Students must be clear about which coefficient they are referring to in order to get a score 
of 2. 

Note: 

Students could describe substitution as a method of finding the slope, but they need to 
give sufficient details to evaluate that method. 

Example: 

The equation is rewritten asy = (-3/5)x + 3. In that form, the term multiplied with the x is 
the slope, so -3/5 is the slope. 

(Because we care more about method and explanation for part b, they can get credit if they 
correctly explained what they did but made a calculation error when they rewrote the equation in 
slope-intercept form. They would miss part a, but we wouldn ’t penalize them in part b as long as 
they give a correct explanation. Also, many students will say the term "next to ” or with the “x”. 

That is fine as long as it ’s clear what they are referring to.) 

Example: 

I rewrote the equation in slope-intercept form, and -3/5 is the slope because it is the “m” 
term in the equation. 

(Students would get credit even if they don ’t specifically mention -3/5 because it’s known that “m " 
is the slope in slope-intercept form.) 

Example: 

I rewrote the equation in terns of y. In the equation y = (-3/5 )x + 3, -3/5 is the slope. 

(If the student talks about solving or rewriting the equation in terms ofy, it’s important that it’s 
evident where the -3/5 came from. In this case, it’s clear because the student gave the equation that 
results when the equation is rewritten in terms ofy.) 

Example: 

I plugged in a 0 and a 1 forx and goty values of 3 and 12/5 fory. Then I used those 
values in this formula for slope, m = yi-yd'ttj-'Xx to find the slope. 

(Student provides values for two points [may include minor calculation errors for one of 
the two points] and formula for using points to calculate slope [or simply set up the 
expression correctly using the two points]) 
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Score 

Description 

Notes 

1 

Student 

gives a 

partially 

correct 

explanation, 

an 

explanation 
that isn’t 
completely 
clear, or an 
incomplete 
explanation 

Example: 

I solved fory and got -3/5. 

(If they say all this but say it equals (-3/5)x instead of -3/5, you can still give them a 1 because they 
are showing some understanding of the process (even though they are not completely correct about 
the slope ) 

Example: 

I wrote the equation in terms ofy and found m. 

(It ’s not clear what they are talking about because “m” only means something if we ’re referring to 
slope-intercept form.) 

Example: 

I rewrote the equation in slope- intercept form and got the slope. 

(It’s unclear how the student got the slope from slope-intercept form.) 

Example: 

I plugged in a 0 and a 1 forx and goty values of 3 and 12/5 fory. Then I used these values 
to calculate the slope. 

(Student knows how to use line equation to find two points, but does not give the formula for 
calculating slope from these two points or show calculation of slope by using these numbers in the 
formula. If the student had done either, the student would have received a 2.) 

Example : 

I plugged in a 0 and a 1 forx and goty values and used y 2 -yi/x 2 -xi 

(Student does not specify the y values or how to calculate them using the equation for the line: 
basically, the student knows that values can be calculated using the line equation and that two 
points with x and y values are needed to calculate slope.) 

Example'. 

1 found two points on the line and used m = y 2 -yi/x 2 -xi 

(Student does not specify they values or how to calculate using the equation for the line.) 

0 

Incorrect, 
too unclear 

to 

understand, 
or too 
incomplete 

Example: 

I solved fory and found the slope. 

Example: 

I rewrote the equation, and -3/5 is the slope. 

Example: 

1 used slope-intercept form. 

Example: 

I plugged in a 0 and a 1 forx and calculated the slope. 


* If students leave a question blank or write ? or IDK, write “BL ” as the score. If students used 
primarily numbers and/or symbols (instead of words) to correctly show their procedure, write “num 
If students incorrectly show their procedure with numbers, count that as a 0. 
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Question 25 


Use these 4 quadratic equations and 4 sketches of the graphs of quadratic equations to answer 
question 25. 

Qy = -x 2 + x+12 © y = (x~6)(x + 1) Q y = (x + 3)(x - 4) ©y = (x-6) 2 +l 



Graph 1 


Graph 2 


Graph 3 


Graph 4 


; 

; 

Equation !A [matches IGraph! [because:! 


Sample Answer : 2, it opens downwards because the value of the a term is negative 


Although this question isn 't split into parts, we’re going to grade this as if there are two parts. 


Rubric for part a): 


Score 

Description 

Notes 

1 

Student correctly writes “2” for Graph 2. 


0 

Incorrect or too unclear. 

Note: Many students used letters instead of numbers for the 
graphs because they got confused. Although they may mean Graph 

2 when they write Graph B, we can ’t assume that. 


* If students leave a question blank or write ? or IDK, write "BL ” as the score. If students do something other than write 2 
(e.g., they draw an arrow to Graph 2), give them credit if their answer is clear but also make a note about it in a notes 
column. 


Rubric for part b): 
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Score 

Description 

Notes 

2 

Student gives a clear explanation. This 
may include: 

Explaining that the figure opens 
downwards because the a term is 
negative (or explaining that there is a 
negative in front of the x 2 term, 
meaning the figure should have a 
maximum/open downwards) 

OR 

Explaining that the figure crosses the 
y-axis at (0,12) because substituting 0 
for x in the equation results in 1 2 for y 
(or explaining that the c tenn in the 
equation is the y-intercept) 

Note: 

To get a 2, it is important that the student explains how they know 
something rather than simply stating how they know something so it s 
clear how they determined their answer. 

Note: 

If a student makes a big mistake with important terminology (e.g., 
explaining that the vertex is at (0,12), he/she cannot receive full credit 
even if the explanation is otherwise correct. 

Note: 

Ideally, the student should relate the graph to the equation and not just 
refer to one or the other (but this may not always be the case). 

1 

Partially correct 

explanation/incomplete explanation or 
an explanation that isn’t completely 
clear 

Example: 

The equation and figure both have ay-intercept at (0,12). 

(Student didn ’t explain how he/she knows that. More explanation would 
be needed to know that the student didn ’t just choose to describe the 
graph.) 

Example: 

1 know the figure should cross the x-axis at (-3,0) and (4,0) because y = - 
x 2 + x + 12 is equivalent to y = -(x-4)(x+3). If you then sety to 0, x equals 

4 and -3. 

(This is a good description, but it is not sufficient for full credit because 
more than one choice includes a graph that crosses at those points.) 

Example: 

The graph I chose is the only one with a maximum. 

(Not explicit enough about why they wanted a graph with a maximum.) 

Example: 

I chose that graph because the sign in front of x 2 is negative. 

(Not explicit enough about why that would lead to a graph that opens 
downwards /has a maximum.) 

0 

Incorrect, too unclear, or too 
incomplete 

Example: 

The equation and figure both intercept the x-axis at (-3,0) and (4,0). 

Note: 

(If a student says he/she graphed the equation to determine which graph 
matched the equation but showed no evidence of graphing, give it a 0.) 

Example: 

It is negative. 

(T oo unclear) 


* If students leave a question blank or write ? or ID K, write “BL ” as the score. If students used primarily numbers and/or 
symbols (instead of words) to correctly show their procedure, write “num ”. If students incorrectly show their procedure 
with numbers, count that as a 0. 
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Appendix B: MDC Teacher Log Descriptives 


Table B1 


Days Spent on Mathematics Activities 


Days spent teaching 

n 

Mean 

SD 

Minimum 

Maximum 

Classroom Challenge 

27 

1.93 

0.57 

1.00 

3.00 

Prior instruction on math concepts 
covered by Classroom Challenge 

27 

4.64 

4.01 

0.50 

15.00 


Table B2 

Extent to Which Teachers Followed the Detailed Lesson Plans 


n 

Mean 

SD 

Minimum 

Maximum 

27 

3.10 

0.23 

2.80 

3.75 


Note. 1 = not at all, 2 = somewhat, 3 = mostly, and 4 = completely. 
Table B3 


Average Proportion of Time Spent on the Different Classroom Challenge Activities 


Activities 

n 

Mean 

SD 

Minimum 

Maximum 

Pre-assessment 

27 

11.24 

3.95 

5.00 

20.00 

Small-group collaborative work 

27 

34.48 

8.66 

20.00 

48.50 

Whole-class discussion 

27 

18.58 

4.93 

10.75 

31.25 

Individual work 

27 

7.53 

4.59 

0.00 

15.00 

Student presentations 

27 

7.41 

4.48 

0.00 

16.00 

Teacher-led review of content/concepts 

27 

9.39 

4.43 

0.00 

19.25 

Post-assessment 

27 

11.08 

3.42 

5.00 

20.00 

Other, please specify 

27 

0.26 

0.65 

0.00 

2.00 


Note. The means represent the average proportion of time spent by each teacher across the study rather than per log. 
Proportions on each log were measured in 10% increments for a total of 100% per log. 
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Table B4 


When Students Completed the Pre-Assessment 


Variable 

n 

Mean 

SD 

Minimum 

Maximum 

In class at start of Classroom Challenge 

27 

0.37 

0.38 

0 

1 

Other 

27 

0.63 

0.38 

0 

1 


Note. Other responses ranged from one day to one week prior to the start of the Classroom Challenge. 


Table B5 


Time Spent Reviewing Students ’ Answers to the Pre-Assessment 


Variable 

n 


Mean 

SD 


Minimum 

Maximum 

0 min 

27 


0.01 

0.04 


0.00 

0.20 

1-5 min 

27 


0.15 

0.28 


0.00 

1.00 

6-15 min 

27 


0.25 

0.37 


0.00 

1.00 

16-30 min 

27 


0.34 

0.36 


0.00 

1.00 

3 1-60 min 

27 


0.19 

0.27 


0.00 

1.00 

>1 hour 

27 


0.04 

0.11 


0.00 

0.50 

Note. Means represent percentages. 





Table B6 








How Reviewed Students ’ Answers to the Pre-Assessment 


Method 


n 

Mean 


SD 

Minimum 

Maximum 

Alone 


27 

0.49 


0.40 

0.00 

1.00 

With a colleague 


27 

0.38 


0.41 

0.00 

1.00 

With the class 


27 

0.06 


0.17 

0.00 

0.75 

Other 


27 

0.05 


0.13 

0.00 

0.50 


Note. Means represent percentages. 
Table B7 


Understanding of Student Misconceptions 


Attitudes 

n 

Mean 

SD 

Minimum 

Maximum 

Understanding comes from studying their 
answers on the pre-assessment. 

27 

2.79 

0.54 

1.00 

3.57 

Difficult to identify from the pre-assessment. 

27 

1.93 

0.39 

1.00 

3.00 


Note. 1 = strongly disagree, 2 = disagree, 3 = agree, and 4= strongly agree. 
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Table B8 


Types of Feedback Given to Students on Their Pre-Assessments 


Feedback type 


n 

Mean 

SD Minimum 

Maximum 

No feedback because Classroom Challenge instruction 
addressed misconceptions 

27 

0.46 

0.39 

0.00 

1.00 

Commented on papers 


27 

0.14 

0.30 

0.00 

1.00 

Graded papers 


27 

0.01 

0.04 

0.00 

0.20 

Wrote questions on papers 


27 

0.06 

0.13 

0.00 

0.50 

Reviewed common errors at start of class 


27 

0.58 

0.35 

0.00 

1.00 

No feedback provided because of time constraints 

27 

0.06 

0.20 

0.00 

1.00 

Note. Responses are not mutually exclusive. 







Table B9 







Techniques to Handle Students Who Struggled During Small-Group Work 




Variable 

n 

Mean 

SD 

Minimum 

Maximum 

Listened but did not intervene 

27 

0.41 

0.36 

0.00 

1.00 


Offered hints and suggestions 

27 

0.45 

0.39 

0.00 

1.00 


Raised questions 

27 

0.93 

0.17 

0.33 

1.00 


Provided the answer 

27 

0.02 

0.06 

0.00 

0.25 


Asked another student to explain concept 

27 

0.78 

0.32 

0.00 

1.00 


Stopped class and reviewed concept 

27 

0.30 

0.37 

0.00 

1.00 



Note. Responses are not mutually exclusive. 
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Table BIO 


Types of Questioning Incorporated Into the Whole-Class Plenary’ 


Variable 

n 

Mean 

SD 

Minimum 

Maximum 

IRE (lnitiate-Respond-Evaluate) : 

(Y es/No questions, questions that ask students to recall facts 
or definitions) 

27 

0.64 

0.38 

0.00 

1.00 

Self-Reflection: 

(Why do you think that? Why is that true? How did you reach 
that conclusion?) 

27 

1.00 

0.00 

1.00 

1.00 

Reflection-on-Others : 

(Do you agree? Does anyone have a different way to explain 
it? Would you ask the rest of the class that question?) 

27 

0.98 

0.08 

0.60 

1.00 

Pattern-Finding and Conjecturing: 

(How did you predict the next case? What is similar and what 
is different about your solution and his/hers? Do you see a 
pattern?) 

27 

0.77 

0.25 

0.00 

1.00 

Mathematical Reasoning: 

(Does that always work? Is that true for all cases? Can you 
think of a counter example?) 

27 

0.84 

0.26 

0.00 

1.00 


Note. 0 = no and 1 = yes. 


Table Bll 


Percentage of Students Who Presented to the Whole Class 


Student presentations to whole class 

n 

Mean 

SD 

Minimum 

Maximum 

Present their solutions 

27 

2.07 

0.51 

1.00 

3.00 

Articulate the reasoning behind their solutions 

27 

2.25 

0.64 

1.00 

4.00 

Note. 1 = <5%, 2 = 5-25%, 3 = 25-50%, 4 = 50- 

Table B12 

When Students Completed the Post-Assessment 

-75%, 5 

= 75-99%, 6 = 100%. 


Variable 

n 

Mean 

SD 

Minimum 

Maximum 

In class at the end of the Classroom Challenge 

27 

0.82 

0.32 

0.00 

1.00 

Homework 

27 

0.01 

0.08 

0.00 

0.40 

Other, please specify 

27 

0.17 

0.31 

0.00 

1.00 


Note. Means represent percentages. 
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Table B 13 

Perceptions About Percentage of Students Who Struggled to Understand the Underlying Mathematical Concepts 


n 

Mean 

SD 

Minimum 

Maximum 

27 

3.02 

0.75 

2.00 

5.00 


Note. 1 = <5%, 2 = 5-25%, 3 = 25-50%, 4 = 50-75%, 5 = 75-99%, 6 = 100%. 

Table B 14 

Perceptions About Percentage of Students Who Gained a Strong Grasp of the Material by the End of Challenge 


n 

Mean 

SD 

Minimum 

Maximum 

27 

3.85 

0.70 

2.33 

5.00 


Note. 1 = <5%, 2 = 5-25%, 3 = 25-50%, 4 = 50-75%, 5 = 75-99%, 6 = 100%. 
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Appendix C: MDC Teacher Survey Descriptives 


Tabled 


Q3: Content Areas Participants Teach 


Content areas 

n 

MDC teachers (%) 

Pre-algebra 

3 

8.3 

Algebra 1 

35 

97.2 

Geometry 

14 

38.9 

Algebra 2 

10 

27.8 

Pre-calculus 

1 

2.8 

Calculus 

0 

0.0 

Probability and statistics 

2 

5.6 

Other 

6 

16.7 


Note, n = 36. 

Table C2 


Q4: Years of Teaching Experience 


Type of experience 

n 

Mean 

SD 

Minimum 

Maximum 

Years of teaching 

36 

8.72 

5.49 

2.00 

21.00 

Years taught in current school 

36 

5.86 

3.33 

2.00 

13.00 

Years taught in current district 

36 

6.96 

4.09 

2.00 

16.00 


Table C3 


Q5-8: Teaching of Different Student Populations 




Yes 


No 

Student population 

n 

% 

n 

% 

ELL students 

23 

63.9 

13 

36.1 

Special education students 

33 

91.7 

3 

8.3 

Students struggling in math 

36 

100.0 

0 

0.0 

Students with advanced mathematics levels 

24 

66.7 

12 

33.3 


Note, n = 36. 
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Table C4 


Q9: Type of Participation in the MDC Initiative 


Type of participation 

n 

MDC teachers (%) 

Required 

23 

63.9 

Voluntary 

12 

33.3 

Other 

1 

2.8 


Note, n = 36. 

Table C5 

Qll-12: MDC Lessons Taught by School Year 


School year 

n 

Mean 

SD 

Minimum 

Maximum 

2011-2012 

31 

4.74 

1.82 

2.00 

8.00 

2012-2013 

35 

5.97 

2.80 

2.00 

12.00 


Table C6 


Q44: MDC Lesson Alignment 


Question 

n 

Mean 

SD 

a. The MDC Lessons align well with my school’s curriculum. 

35 

2.37 

0.69 

b. The MDC Lessons help prepare my students for the current state assessment(s). 

35 

2.09 

0.82 

c. The MDC Lessons align with the CCSS. 

35 

2.57 

0.61 

d. I see the unique value of the MDC Lessons to address the CCSS. 

35 

2.31 

0.53 


Note. 0 = disagree, 1 = disagree somewhat, 2 = agree somewhat, and 3 = agree. 


Table C7 


Q28: Fidelity to the MDC Lessons Teacher Guide 


Extent followed 

n 

MDC teachers (%) 

Completely 

5 

14.3 

Mostly 

28 

80.0 

Somewhat 

2 

5.7 


Note, n = 35. 
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Table C8 

Q29: Class Periods Spent Implementing a Typical MDC Lesson (2012-2013) 


Question n 

Mean 

SD 

Minimum 

Maximum 

Class periods 35 

3.31 

3.64 

1.00 

18.00 


Table C9 


Q33: Practice Used When Assigning Students to Collaborative Groups 


Practice 

n 

MDC teachers (%) 

Students choose their own partners/groups 

8 

22.9 

Assign students to homogeneous math ability groups 

20 

57.1 

Assign students to heterogeneous math ability groups 

7 

20.0 

Note, n — 35. 



Table CIO 



Q35-36: Use of Post-assessments During Most Recent MDC Lessons Taught 

Post assessments 

n 

MDC teachers (%) 

When students completed the post-assessment (« = 35) 



In class, at the end of the plenary/class discussion 

31 

88.6 

As homework 

0 

0.0 

Did not use the post-assessment 

4 

11.4 

Reasons didn’t use post-assessment (n = 4) 



Ran out of time 

2 

50.0 

Seemed redundant 

0 

0.0 

Students understood the material 

0 

0.0 

Too hard 

0 

0.0 

Reason - Other 

2 

50.0 


Note. Only teachers who indicated that they did not use the post-assessment were asked to state a reason. 
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Table Cl 1 


Q1 7: Teacher Perceptions About the Effectiveness of Strategies for Strengthening Mathematical Understanding 



Question 

n 

Mean 

SD 

a. 

Teacher taking on the role of “facilitator” or “coach” 

35 

2.83 

0.38 

b. 

Peer-to-peer problem solving 

35 

2.77 

0.43 

c. 

Asking students guiding questions 

35 

2.91 

0.37 

d. 

Providing class time for students to persevere through difficult math problems 

35 

2.66 

0.68 


Note. 0 = disagree, 1 = disagree somewhat, 2 = agree somewhat, and 3 = agree. 


Table C 12 


Q19: Teacher Perceptions of Whether MDC Lessons Helped Them 





Yes 


No 


Question 

n 

% 

n 

% 

a. 

Find effective strategies for teaching my subject content 

32 

88.9 

4 

11.1 

b. 

Learn new ways to include FA in my classes 

32 

88.9 

4 

11.1 

c. 

Learn detailed info about students’ math strengths and weaknesses 

29 

80.6 

7 

19.4 

d. 

Provide students with more detailed feedback about their work 

25 

69.4 

11 

30.6 

e. 

Implement the CCSS 

28 

77.8 

8 

22.2 

f. 

Create an environment that promotes math discourse 

31 

86.1 

5 

13.9 

g- 

Better engage students 

27 

75.0 

9 

25.0 


Note, n = 36. 

Table C 13 


Q18: Teacher Perceptions of the Effectiveness of MDC Lessons 



Question 

n 

Mean 

SD 

a. 

Improve students’ ability to think mathematically 

35 

2.63 

0.55 

b. 

Provide a curricular resource for teachers in addressing CCSS 

35 

2.43 

0.56 

c. 

Encourage teachers to adjust pedagogy in math from focus on 
process to building conceptual knowledge 

35 

2.69 

0.47 

d. 

Make instruction more engaging for students 

35 

2.57 

0.56 

e. 

Use FA to identify student strengths and weaknesses and to 
inform instruction 

35 

2.60 

0.60 


Note. 0 = disagree, 1 = disagree somewhat, 2 = agree somewhat, and 3 = agree. 
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Table C 14 


Q39: Potential Barriers to Use of MDC Lessons 



Barriers 

n 

Mean 

SD 

a. 

Had sufficient time to plan for lessons 

35 

2.31 

0.68 

b. 

Felt adequately prepared to effectively use the lessons 

35 

2.34 

0.72 

c. 

Had sufficient time to prepare for the pre-assessment 

35 

2.46 

0.70 

d. 

Using MDC takes too much time away from covering required 
curriculum topics 

35 

1.49 

0.85 

e. 

Prep required for the MDC collaborative activity is an impediment to 
using the lesson 

35 

1.71 

0.86 

f. 

During the collaborative portion of MDC, it is difficult to interact with 
every group 

35 

1.49 

0.92 


Note. 0 = disagree, 1 — disagree somewhat, 2 = agree somewhat, and 3 = agree. 


Table C15 


Q16: Individuals Who Visited Teachers’ Classrooms During MDC Lessons 




Yes 


No 

Individuals 

n 

% 

n 

% 

a. District or network MDC project lead 

25 

71.4 

10 

28.6 

b. Principal 

14 

40.0 

21 

60.0 

c. Instructional coach/department head 

27 

77.1 

8 

22.9 

d. Teacher colleague 

33 

94.3 

2 

5.7 


Note, n = 35. 
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Table C16 


Q43: Administration and MDC Lessons 



Question 

n 

Mean 

SD 

School administrators 




a. 

Have a firm understanding of the MDC Lessons 

31 

1.61 

0.99 

b. 

Have made FA a priority at my school 

34 

2.18 

0.83 

c. 

Encouraged me to participate in MDC 

34 

2.44 

0.82 

d. 

Provided me with feedback about my instruction of MDC Lessons 

34 

1.32 

1.22 

e. 

Concerned that the MDC Lessons are taking time away from other 
instructional priorities 

33 

0.52 

0.91 

f. 

Attended PD about the MDC Lessons 

23 

1.13 

0.97 

g- 

Communicated how the MDC lessons are aligned with other school 
initiatives 

26 

1.58 

1.14 

District administrators 




h. 

Support the MDC initiative 

32 

2.68 

0.54 


Note. 0 = disagree, 1 = disagree somewhat, 2 = agree somewhat, and 3 = agree. 


Table Cl 7 

Q45: Teacher Collaboration During MDC Implementation 


Yes No 

Teacher collaboration n % n % 

Regularly scheduled common planning time with colleagues to discuss MDC 21 60.0 14 40.0 

Note, n = 35. 

Table C 18 

Q47-48: MDC Meetings to Discuss Student Work, Instructional Strategies, or Teaching Approaches 


Frequency 

Scheduled 

Informal 

n 

% 

n 

% 

At least once a week 

2 

9.5 

15 

42.9 

Every other week 

4 

19.0 

12 

34.3 

Once a month 

8 

38.1 

3 

8.6 

Once per quarter/trimester/semester 

7 

33.3 

5 

14.3 


Note. Scheduled meetings (n = 21), informal meetings (« = 35). 
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Table C19 


Q46 & Q49: Perceptions of Teacher Collaboration During MDC Implementation 



Question 

n 

Mean 

SD 

Teacher collaboration 

Describes MDC colleagues as collaborative 

35 

2.66 

0.59 

Benefits of collaboration with MDC colleagues 

a. More effectively use the MDC FA strategies 

35 

2.71 

0.46 

b. 

Better support student learning 

35 

2.57 

0.66 

c. 

Develop the MDC lessons in my class 

35 

2.54 

0.70 

d. 

Review pre-assessments 

35 

2.54 

0.70 

e. 

Facilitate collaborative group work 

35 

2.57 

0.66 

f. 

Facilitate the plenary or whole-class discussion 

35 

2.46 

0.66 

g- 

Identify common math misconceptions 

35 

2.63 

0.65 

h. 

Determine where to use an MDC lesson in a unit 

35 

2.80 

0.41 

i. 

Develop feedback questions 

35 

2.69 

0.63 

j- 

Determine how to group students 

35 

1.91 

1.04 

k. 

Review post-assessments 

35 

2.51 

0.78 


Note. 0 = disagree, 1 = disagree somewhat, 2 = agree somewhat, and 3 = agree. 


Table C20 


Q53: Teacher Participation in MDC Professional Development 



Yes 

No 

Question 

n % 

n % 

Participated during the 2012-2013 school year 

27 75.0 

9 25.0 


Note, n = 36. 

Table C21 


Q55: Level of Participation in Formal MDC Professional Development Sessions 


Question 

n 

Mean 

SD 

Minimum 

Maximum 

Sessions attended during the 2012-2013 school year 

27 

2.93 

1.52 

1.00 

8.00 
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Table C22 


Q58: Content Included in MDC Professional Development Sessions in Which Teachers Participated 





Yes 


No 


Question 

n 

% 

n 

% 

a. 

Using MDC Lessons as a way to implement the CCSS 

18 

66.7 

9 

33.3 

b. 

Administering the pre-assessment 

21 

77.8 

6 

22.2 

c. 

Identifying common misconceptions through the pre-assessment results 

26 

96.3 

1 

3.7 

d. 

Developing feedback questions 

27 

100.0 

0 

0.0 

e. 

Facilitating small group work with guiding questions 

25 

92.6 

2 

7.4 

f. 

Facilitating whole group plenary discussions 

22 

81.5 

5 

18.5 

g- 

Placing lessons at the appropriate point in the unit 

19 

70.4 

8 

29.6 

h. 

Determining how to use the post-assessment results to guide continued 
instruction in the unit 

19 

70.4 

8 

29.6 

i. 

Differentiating MDC lesson instruction to meet student needs 

14 

51.9 

13 

48.1 

j- 

Implementing MDC lessons with ELL students 

4 

14.8 

23 

85.2 

k. 

Implementing MDC lessons with special education students 

9 

33.3 

18 

66.7 

1 . 

Implementing MDC lessons with students who are struggling with math 

15 

55.6 

12 

44.4 

m. 

Implementing MDC lessons with students with advanced math levels 

12 

44.4 

15 

55.6 


Note, n = 27. 

Table C23 


Q56-57: Types and Perceived Effectiveness of Professional Development in Which Teachers Participated 






If participated, perceived effectiveness 



Yes, participated 


Yes 


No 


Type 

n 

% 

n 

% 

n 

% 

a. 

One-on-one classroom visits 

20 

74.1 

18 

90.0 

2 

10.0 

b. 

Coaching 

22 

81.5 

22 

100.0 

0 

0.0 

c. 

Webinars 

2 

7.4 

1 

50.0 

1 

50.0 

d. 

Small group meetings 

25 

92.6 

25 

100.0 

0 

0.0 

e. 

Schoolwide meetings 

23 

85.2 

20 

87.0 

3 

13.0 

f. 

Districtwide meetings 

23 

85.2 

17 

73.9 

6 

26.1 

g- 

Cross-district meetings 

11 

40.7 

9 

81.8 

2 

18.2 


Note. Only teachers who indicated that they participated in a specific type of PD were asked about the effectiveness. 
n — 27. 
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Table C24 


Q25: Student Engagement During MDC Lessons 



MDC teachers 

Effect on engagement 

n 

% 

More engaged 

14 

38.9 

Same level of engagement 

18 

50.0 

Less engaged 

3 

8.3 


Note, n = 54. 

Table C25 


Q26: Teacher Perceptions on Impact on Students in 

General 



Impact on students 

n 

Mean 

SD 

a. Improved my students’ mathematical reasoning 

35 

2.17 

0.71 

b. Supporting my students’ college-readiness 

35 

2.17 

0.71 


Note. 0 = disagree, 1 = disagree somewhat, 2 = agree somewhat, and 3 = agree. 


Table C26 


Q21: Teacher Perceptions on Impact on Students During Last MDC Lesson 




Yes 


No 

Impact on students 

n 

% 

n 

% 

a. Majority of students improved their content knowledge 

29 

82.9 

6 

17.1 

b. Majority of students improved their conceptual understanding 

29 

82.9 

6 

17.1 


Note, n = 35. 
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Appendix D: Analysis of MDC Student Work Artifacts 


Table D1 

Goals and Standards for FALs Included in MDC Analysis 


FAL 


Goals 


Standards 


Practices Items 


FAL03 • Solving a problem using two linear 
equations with two variables. 

• Interpreting the meaning of 
algebraic expressions. 


Algebra: 

• CED: Create equations 
that describe numbers or 
relationships. 


• REI: Solve systems of 
equations. 


2. Reason abstractly 
and quantitatively. 

3. Construct viable 
arguments and 
critique the 
reasoning of others. 


Accuracy: 3 
Explanation: 4 
Misconceptions: 4 


FAL13 • Recognize the differences between 
equations and identities. 

• Substitute numbers into algebraic 
statements in order to test their 
validity in special cases. 

• Resist common errors when 


Algebra: 

• SSE: Interpret the 
structure of expressions. 
Write expressions in 
equivalent forms to solve 
problems. 


3. Construct viable 
arguments and 
critique the 
reasoning of others. 

7. Look for and make 
use of structure. 


Accuracy: 10 
Explanation: 1 
Misconceptions: 6 


manipulating expressions such as 

2(x - 3) = 2x - 3; (x + 3) 2 = x 2 + 3 2 . 

• Carry out correct algebraic 
manipulations. 

• REI: Solve equations and 
inequalities in one 
variable. 



FAL 16 • Recognizing the order of algebraic 

operations. 

• Recognizing equivalent 
expressions. 

• Understanding the distributive laws 
of multiplication and division over 
addition (expansion of 
parentheses). 

Algebra: 

• SSE: Interpret the 
structure of expressions. 

• APR: Rewrite rational 
expressions. 

2. Reason abstractly 
and quantitatively. 

7. Look for and make 
use of structure. 

Accuracy: 9 
Explanation: 3 
Misconceptions: 1 

FAL22 • Find, from their equations, lines 

Geometry: 

1 . Make sense of 

Accuracy: 2 


that are parallel and perpendicular. 

• Identify and use intercepts. 

• It also aims to encourage 
discussion on some common 
misconceptions about equations of 
lines. 


• PE: Use coordinates to 
prove simple geometric 
theorems algebraically. 

Functions: 

• IF : Analyze functions 
using different 
representations. 


problems and 
persevere in solving 
them. 

3. Construct viable 
arguments and 
critique the 
reasoning of others. 

7. Look for and make 
use of structure. 


Explanation: 1 
Misconceptions: 3 
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Appendix E: Quasi-Experimental Analysis of MDC Effects 


Table El 


MDC Effects on ACT PLAN Total Scores, Including Interaction Effects (Models 1 and 2) 


ACT PLAN mathematics (total) 

Model 1 coefficient (SE) 

Model 2 coefficient (SE) 

MDC treatment 

0.092 (0.031)* 

0.110 (0.022)* 

School effectiveness 

0.284 (0.092)* 

0.300 (0.093)* 

Teacher effectiveness 

0.199(0.101) 

0.154 ( 0.094) 

MDC treatment by teacher effectiveness 
interaction 

0.251 (0.173) 

0.285 (0.158) 

Treatment by demographic characteristic 
interactions 



Gender 

0.002 (0.026) 

0.00 (0.029) 

Special education 

0.042 (0.034) 

0.055 (0.037) 

Free/reduced price lunch eligible 

0.013 (0.028) 

0.012 (0.027) 

Prior achievement 

0.040 (0.021) 

0.023 (0.017) 


Note. Fixed effects for demographic predictors not shown. 
*p < -05. **p < .01. 
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Table E2 


Models 1 and 2 With Demo Interactions: Final Estimation of Fixed Effects on Math PLAN Algebra Score With 
Robust Standard Errors 


Fixed effect 

Model 1 coefficient (SE) 

Model 2 coefficient (SE) 

Level 1 variables 



Female 

0.022 (0.015) 

0.022 (0.018) 

White 

-0.058 (0.050) 

-0.058 (0.053) 

Flispanic 

-0.087 (0.064) 

-0.126 (0.066) 

Black 

0.021 (0.057) 

0.013 (0.061) 

Asian 

0.000 (0.075) 

-0.019 (0.087) 

English language learner 

-0.114 (0.069) 

-0.073 (0.074) 

Special education 

0.019 (0.026) 

0.032 (0.026) 

Free/reduced price lunch eligible 

-0.003 (0.015) 

-0.002 (0.015) 

Prior achievement 

0.434 (0.008)** 

0.435 (0.008)** 

Level 2 variables 



MDC treatment 

0.128 (0.035)** 

0.130 (0.026)** 

School effectiveness 

0.273 (0.110)* 

0.309 (0.103)** 

Teacher effectiveness 

0.113 (0.094) 

0.060 (0.087) 

Missing teacher effectiveness 

0.018 (0.022) 

0.014 (0.020) 

MDC treatment by teacher effectiveness 
interaction 

0.342 (0.197) 

0.420 (0.178)* 

Treatment by demographic characteristic 
interactions 



Female 

-0.010 (0.030) 

0.005 (0.036) 

Special education 

0.040 (0.040) 

0.070 (0.045) 

Free/reduced price lunch eligible 

0.019 (0.029) 

0.027 (0.029) 

Prior achievement 

0.041 (0.017)* 

0.030 (0.016) 


*p < .05. **£><.01. 
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